PDA

View Full Version : Linux Makefiles - File name manipulations...



KurtEck
08-17-2015, 09:12 AM
Right now I am in the process of hacking up some of the HR-OS1 makefiles, as I have already been bit a few times, by things not recompiling when things they depend on are updated. I have a first pass at it up on my fork up on github in my Joystick fork... It is working better, but I would still like to clean it up some, as for example I don't like that it scatters all of the dependency and object files in all of the source file locations. Also I would like to add the option the ability to make either retail or debug binaries without changing makefile and maybe have them both be able to be there at the same time...

For example, Right now, the library (HROS1-framework/Linux/build) makefile paths looks like:

#~~~~~~~~~~~~~~~~~~~~ Source Files ~~~~~~~~~~~~~~~~~~~~
SOURCES = \
../../Framework/src/CM730.cpp \
../../Framework/src/math/Matrix.cpp \
../../Framework/src/math/Plane.cpp \
../../Framework/src/math/Point.cpp \
../../Framework/src/math/Vector.cpp \
../../Framework/src/math/QuadraticStateTransform.cpp \
../../Framework/src/math/MotionState.cpp \
../../Framework/src/motion/JointData.cpp \
../../Framework/src/motion/Kinematics.cpp \
../../Framework/src/motion/MotionManager.cpp \
../../Framework/src/motion/MotionStatus.cpp \
../../Framework/src/motion/AngleEstimator.cpp \
../../Framework/src/motion/modules/Action.cpp \
../../Framework/src/motion/modules/Head.cpp \
../../Framework/src/motion/modules/Walking.cpp\
../../Framework/src/vision/BallFollower.cpp \
../../Framework/src/vision/PS3BallFollower.cpp \
../../Framework/src/vision/LineFollower.cpp \
../../Framework/src/vision/RobotFollower.cpp \
../../Framework/src/vision/ConnectRegions.cpp \
../../Framework/src/vision/RadonTransform.cpp \
../../Framework/src/vision/BallTracker.cpp \
../../Framework/src/vision/ColorFinder.cpp \
../../Framework/src/vision/Image.cpp \
../../Framework/src/vision/ImgProcess.cpp \
../../Framework/src/controller/JoystickController.cpp \
../../Framework/src/commander/SerialInputCommander.cpp \
../../Framework/src/minIni/minIni.cpp \
LinuxActionScript.cpp \
LinuxCamera.cpp \
LinuxCM730.cpp \
LinuxMotionTimer.cpp \
LinuxNetwork.cpp


#~~~~~~~~~~~~~~~~~~~~ Object Files ~~~~~~~~~~~~~~~~~~~~
OBJS:= $(subst .cpp,.o,$(SOURCES))


DEPS:= $(subst .cpp,.d,$(SOURCES))


So after the makefile is run there are objects (and dependencies) are scattered in many different source directories.

What I would like to do is to update the OBJS: and DEPS: to instead, create a directory under build, either retail or debug
with all of the files.

I know it can be done, and I am just about to start playing with $(notdir and $(basename and $(addsuffix and $(addprefix
to see if I can build the right strings. But was wondering if someone had a clean way to do this, as I am pretty sure that this is common thing.

Thanks
Kurt

tician
08-17-2015, 09:51 AM
The following only works when the source files are in makefile's root folder or its subfolders. Variations of this have worked well enough for simpler linux and avr programs, but could always be better if I bothered to learn more about makefiles (thinking the order of some of the flags/options might be wrong for newer versions of gcc). The avr version has lots of other additions for hex and eep file generation as well as generating dependencies for files (gets put into .dep in makefile root folder). Would prefer moving to CMake, but too stoopid to spend time learning it...

The makefile reads through all source files listed in SRCS and filters for C and C++ files to give them their corresponding object extension (and ensure correct compiler used). RAWOBJS is simply the combination of the C and C++ objects (SRCS with new extensions), then TARGET_OBJS makes sure all the sources are built into objects in an 'obj' folder in the root of the makefile to keep the source folders clean of binaries and make it easier to clean (just delete the 'bin' and 'obj' folders).



#~~~~~~~~~~~~~~~~~~~~ Output File Name ~~~~~~~~~~~~~~~~~~~~
PROJECT_NAME = yetis_ltb_v0

#~~~~~~~~~~~~~~~~~~~~ Source Files ~~~~~~~~~~~~~~~~~~~~
SRCS = \
yetis_ltb_v0.cpp \
usiTwiSlave.c

#~~~~~~~~~~~~~~~~~~~~ Object Files ~~~~~~~~~~~~~~~~~~~~
BINDIR = bin
OBJDIR = obj

TARGET = $(BINDIR)/$(PROJECT_NAME)

CSRCS = $(filter %.c,$(SRCS))
CPPSRCS = $(filter %.cpp,$(SRCS))
RAWOBJS = $(CSRCS:.c=.c.o) $(CPPSRCS:.cpp=.cpp.o)
TARGET_OBJS = $(addprefix $(OBJDIR)/,$(RAWOBJS))

#~~~~~~~~~~~~~~~~~~~~ Include Directories ~~~~~~~~~~~~~~~~~~~~
INCLUDE_DIRS = -I. -I./inc

#~~~~~~~~~~~~~~~~~~~~ Library Directories ~~~~~~~~~~~~~~~~~~~~
LIBRARY_DIRS = -L.
LIBS =

#~~~~~~~~~~~~~~~~~~~~ Compiler Options ~~~~~~~~~~~~~~~~~~~~
COMPILE_OPTS = -g -O3 -fno-common
COMPILE_OPTS += -Wall -Wextra -Wmain -pedantic -pedantic-errors -Wshadow -Winit-self -Wredundant-decls -Wcast-align -Wundef -Wfloat-equal -Winline -Wunreachable-code -Wmissing-declarations -Wmissing-include-dirs -Wswitch-enum -Wswitch-default

#~~~~~~~~~~~~~~~~~~~~ Linker Options ~~~~~~~~~~~~~~~~~~~~
LINKER_OPTS = -s -ldl -lrt

#~~~~~~~~~~~~~~~~~~~~ Toolchain Prefix ~~~~~~~~~~~~~~~~~~~~
TCHAIN_PREFIX=

CC = $(TCHAIN_PREFIX)gcc
CFLAGS = $(INCLUDE_DIRS) $(COMPILE_OPTS)

CXX = $(TCHAIN_PREFIX)g++
CXXFLAGS = $(INCLUDE_DIRS) $(COMPILE_OPTS)

AS = $(TCHAIN_PREFIX)gcc
ASFLAGS = $(COMPILE_OPTS) -c

LD = $(TCHAIN_PREFIX)gcc
LDFLAGS = $(INCLUDE_DIRS) $(LIBRARY_DIRS) $(LIBS) $(LINKER_OPTS)

OBJCP = $(TCHAIN_PREFIX)objcopy
OBJCPFLAGS_HEX = -O ihex
OBJCPFLAGS_BIN = -O binary

OBJDUMP = $(TCHAIN_PREFIX)objdump
OBJDUMPFLAGS = -h -S -C -D

SIZE = $(TCHAIN_PREFIX)size

AR = $(TCHAIN_PREFIX)ar
ARFLAGS = cr


#~~~~~~~~~~~~~~~~~~~~ messages ~~~~~~~~~~~~~~~~~~~~
NO_COLOR=\x1b[0m
OK_COLOR=\x1b[32;01m
ERROR_COLOR=\x1b[31;01m
WARN_COLOR=\x1b[33;01m
OK_STRING=$(OK_COLOR)[OK]$(NO_COLOR)
ERROR_STRING=$(ERROR_COLOR)[ERRORS]$(NO_COLOR)
WARN_STRING=$(WARN_COLOR)[WARNINGS]$(NO_COLOR)

MSG_ERRORS_NONE = Errors: none
MSG_BEGIN = -------- begin --------
MSG_END = -------- end --------
MSG_BUILD_DIRS = Checking build directories exist:
MSG_SIZE_BEFORE = Size before:
MSG_SIZE_AFTER = Size after:
MSG_FLASH = Creating load file for Flash:
MSG_EEPROM = Creating load file for EEPROM:
MSG_EXTENDED_LISTING = Creating Extended Listing:
MSG_SYMBOL_TABLE = Creating Symbol Table:
MSG_LINKING = Linking:
MSG_COMPILING = Compiling C:
MSG_COMPILING_CPP = Compiling C++:
MSG_ASSEMBLING = Assembling:
MSG_CLEANING = Cleaning project:
MSG_CREATING_LIBRARY = Creating library:


#~~~~~~~~~~~~~~~~~~~~ all ~~~~~~~~~~~~~~~~~~~~
all: begin gccversion spotcheck sizebefore build sizeafter end

#~~~~~~~~~~~~~~~~~~~~ build ~~~~~~~~~~~~~~~~~~~~
spotcheck:
@echo $(MSG_BUILD_DIRS)
test -d $(BINDIR) || mkdir -p $(BINDIR)
test -d $(OBJDIR) || mkdir -p $(OBJDIR)
@echo

build: $(TARGET)

$(TARGET): $(OBJS)
$(CXX) -o [email protected] $(OBJS) $(LDFLAGS)

%.c.o: $(CSRCS)
$(CC) $(CFLAGS) -c $< -o [email protected]

%.cpp.o: $(CPPSRCS)
$(CXX) $(CXXFLAGS) -c $< -o [email protected]

#~~~~~~~~~~~~~~~~~~~~ Eye candy ~~~~~~~~~~~~~~~~~~~~
begin:
@echo
@echo $(MSG_BEGIN)

end:
@echo $(MSG_END)
@echo

sizebefore:
@if test -f $(TARGET); then echo; \
echo $(MSG_SIZE_BEFORE); \
$(SIZE) $(TARGET); \
2>/dev/null; echo; fi

sizeafter:
@if test -f $(TARGET); then echo; \
echo $(MSG_SIZE_AFTER); \
$(SIZE) $(TARGET); \
2>/dev/null; echo; fi

gccversion :
@$(CC) --version

#~~~~~~~~~~~~~~~~~~~~ clean ~~~~~~~~~~~~~~~~~~~~
clean: begin clean_list end

clean_list:
rm -rf $(OBJDIR)
rm -rf $(BINDIR)

#~~~~~~~~~~~~~~~~~~~~ backup ~~~~~~~~~~~~~~~~~~~~
backup: clean
tar cJvf ../$(PROJECT_NAME)_`date +"%Y-%m-%d_%H%M"`.tar.xz *

jwatte
08-17-2015, 11:13 AM
The best way to do dependency tracking is to make the compiler do it!
First, make sure you pass "-MMD" to the compiler when compiling.
This will generate a ".d" file right next to the output ".o" file.
Then, make sure you include all these .d files in your makefile.
Because they're not there after a make clean (or before first build,) you have to use a "include ignoring errors" directive.

Typically, that looks like this at the end of the makefile:


-include $(patsubst %.o,%.d,$(OBJS))

KurtEck
08-17-2015, 11:46 AM
Thanks guys, so playing with it.

I believe my stuff I am working with is from earlier stuff I borrowed from tician a few years ago, which
from my Raspberry Pi project has stuff like this at the end:


#~~~~~~~~~~~~~~~~~~~~ Dependency Generation
include $(subst .cpp,.d,$(SOURCES))

%.d: %.cpp
$(CC) -M $(CXXFLAGS) $< > [email protected]$$$$; \
sed 's,\($*\)\.o[ :]*,\1.o [email protected] : ,g' < [email protected]$$$$ > [email protected]; \
rm -f [email protected]$$$$

which has been working well for me with that project as the makefile is in same directory as sources and objects. But i am now trying to move stuff to objects directory and the sources are scattered:

This works when the sources are not scattered:

$(BUILDDIR)%.d: %.cpp
$(CC) -M $(CXXFLAGS) $< > [email protected]$$$$; \
sed 's,\($*\)\.o[ :]*,\1.o [email protected] : ,g' < [email protected]$$$$ > [email protected]; \
rm -f [email protected]$$$$

But as soon as it gets one of the files like: ../../Framework/src/minIni/minIni.cpp
It fails. So I think I need to generate more explicit dependency

Or I can just live with the binaries and dependency files being scattered.

tician
08-17-2015, 12:26 PM
The avr makefile came with:


# Compiler flags to generate dependency files.
GENDEPFLAGS = -MMD -MP -MF .dep/$(@F).d

and $(GENDEPFLAGS) added to CFLAGS, CXXFLAGS, and ASFLAGS prior to the $(INCLUDE_DIRS)

Then near the end of the file has:


# Include the dependency files.
-include $(shell mkdir .dep 2>/dev/null) $(wildcard .dep/*)


No idea what most of the compiler options do.


Since the DARwIn-OP/HROS-1/5 Framework is technically a library linked by custom programs, why not simply re-arrange things...


>humanoids
->lib
-->blobs
-->HROS1
--->makefile (moved from build)
--->bin, objs (temporary)
--->Framework
---->include, src
--->Linux
---->include, src (formerly build)
->projects

It would eliminate the need for the '../../' that causes compiler problems and better isolate the standard library code from custom user code utilizing the library. Can have the library makefile build everything in local 'obj' and 'bin' folders then copy/move the final binary file up to '../blobs/hros1.whatever'. It is very easy to add precompiled libraries to linker options with '../../lib/blobs/hros1' from a makefile in '/humanoids/projects/demo', and can have project makefiles call the desired library makefile to ensure the binary is up to date.

KurtEck
08-17-2015, 02:00 PM
Thanks, makes a lot of sense to rearrange the project, which I would be very inclined to try.

But since I am not the owner of the primary github project, I should probably get their input on this before I try it out.

Thanks again

jwatte
08-17-2015, 08:02 PM
Minor nits about the alternative methods suggested above:
You should never need a separate step for generating the .d files -- they are generated as a side effect of creating the .o files. (And if there are no .o files, well then the .o files absolutely need to be built!)
You should not include "all .d files" in some directory, because that will start causing problems when you rename / delete source files.

tician
08-17-2015, 08:26 PM
Made a branch with rearranged/renamed layout (https://github.com/tician/HROS1-Framework/tree/tician-reform) and fixed several warnings and errors from gcc-5.1.0 (and using -std=gnu++0x) (mostly attempts to create int or char arrays with variable lengths not using malloc or new; or replacing 'static const' with 'static constexpr' for floats). Still lots more float-equals warnings and freaky formatting to clean up, but hros1.a appears to compile everything and archive correctly. No idea if the actual ps3_demo works since I don't have the hardware to test it, but everything appears to compile and link correctly. Also added a check in the ps3_demo makefile for hros1.a already existing so it won't recompile everything in hros1.a every single time.

KurtEck
08-17-2015, 09:59 PM
Thanks Tician,

I just tried it out on my RPI2 and:



[email protected] ~/HROS1-Framework-tician/src/hros1 $ make
make[1]: Entering directory '/home/pi/HROS1-Framework-tician/src/hros1/_blobs'

-------- begin --------
gcc (Debian 4.6.3-14+rpi1) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Checking build directories exist:
test -d obj || mkdir -p obj

/home/pi/HROS1-Framework-tician/src/hros1
framework/src/minIni/minIni.c
framework/src/CM730.cpp framework/src/math/Matrix.cpp framework/src/math/Plane.cpp framework/src/math/Point.cpp framework/src/math/Vector.cpp framework/src/math/QuadraticStateTransform.cpp framework/src/math/MotionState.cpp framework/src/motion/JointData.cpp framework/src/motion/Kinematics.cpp framework/src/motion/MotionManager.cpp framework/src/motion/MotionStatus.cpp framework/src/motion/AngleEstimator.cpp framework/src/motion/modules/Action.cpp framework/src/motion/modules/Head.cpp framework/src/motion/modules/Walking.cpp framework/src/vision/BallFollower.cpp framework/src/vision/PS3BallFollower.cpp framework/src/vision/LineFollower.cpp framework/src/vision/RobotFollower.cpp framework/src/vision/ConnectRegions.cpp framework/src/vision/RadonTransform.cpp framework/src/vision/BallTracker.cpp framework/src/vision/ColorFinder.cpp framework/src/vision/Image.cpp framework/src/vision/ImgProcess.cpp framework/src/controller/PS3Controller.cpp framework/src/commander/SerialInputCommander.cpp linux/src/LinuxActionScript.cpp linux/src/LinuxCamera.cpp linux/src/LinuxCM730.cpp linux/src/LinuxMotionTimer.cpp linux/src/LinuxNetwork.cpp
framework/src/minIni/minIni.c.o framework/src/CM730.cpp.o framework/src/math/Matrix.cpp.o framework/src/math/Plane.cpp.o framework/src/math/Point.cpp.o framework/src/math/Vector.cpp.o framework/src/math/QuadraticStateTransform.cpp.o framework/src/math/MotionState.cpp.o framework/src/motion/JointData.cpp.o framework/src/motion/Kinematics.cpp.o framework/src/motion/MotionManager.cpp.o framework/src/motion/MotionStatus.cpp.o framework/src/motion/AngleEstimator.cpp.o framework/src/motion/modules/Action.cpp.o framework/src/motion/modules/Head.cpp.o framework/src/motion/modules/Walking.cpp.o framework/src/vision/BallFollower.cpp.o framework/src/vision/PS3BallFollower.cpp.o framework/src/vision/LineFollower.cpp.o framework/src/vision/RobotFollower.cpp.o framework/src/vision/ConnectRegions.cpp.o framework/src/vision/RadonTransform.cpp.o framework/src/vision/BallTracker.cpp.o framework/src/vision/ColorFinder.cpp.o framework/src/vision/Image.cpp.o framework/src/vision/ImgProcess.cpp.o framework/src/controller/PS3Controller.cpp.o framework/src/commander/SerialInputCommander.cpp.o linux/src/LinuxActionScript.cpp.o linux/src/LinuxCamera.cpp.o linux/src/LinuxCM730.cpp.o linux/src/LinuxMotionTimer.cpp.o linux/src/LinuxNetwork.cpp.o

gcc -MMD -MP -MF .dep/minIni.c.o.d -I. -I/home/pi/HROS1-Framework-tician/src/hros1/framework/include -I/home/pi/HROS1-Framework-tician/src/hros1/linux/include -g -O2 -fno-common -std=gnu++0x -Wall -Wextra -Wmain -pedantic -pedantic-errors -Wshadow -Winit-self -Wredundant-decls -Wcast-align -Wundef -Wfloat-equal -Winline -Wunreachable-code -Wmissing-declarations -Wmissing-include-dirs -Wswitch-enum -Wswitch-default -fPIC -shared -DLINUX -D_GNU_SOURCE -Wall -g -c /home/pi/HROS1-Framework-tician/src/hros1/framework/src/minIni/minIni.c -o obj/minIni.c.o
cc1: warning: command line option ‘-std=gnu++0x’ is valid for C++/ObjC++ but not for C [enabled by default]
/home/pi/HROS1-Framework-tician/src/hros1/framework/src/minIni/minIni.c:88:16: error: comma at end of enumerator list [-pedantic]
/home/pi/HROS1-Framework-tician/src/hros1/makefile:196: recipe for target 'framework/src/minIni/minIni.c.o' failed
make[1]: *** [framework/src/minIni/minIni.c.o] Error 1
make[1]: Leaving directory '/home/pi/HROS1-Framework-tician/src/hros1/_blobs'
target.mk:26: recipe for target '_blobs' failed
make: *** [_blobs] Error 2
[email protected] ~/HROS1-Framework-tician/src/hros1 $

Git says I am up to date. Will look more in the morning!

Thanks again

Kurt

tician
08-17-2015, 10:40 PM
Apparently an extra, trailing comma in an enum doesn't cause 5.1.0 to stop with an error when using -pedantic, but will cause 4.6.3 to stop with an error. Deleted the extraneous comma from minIni.c and hopefully fixed the symlink issue (would add a symlink to each src/.../include within itself if symlink already exists in 'HROS1-Framework/include').

KurtEck
08-18-2015, 08:37 AM
Thanks again,

Maybe I should have put this under the HROS1 forum.

I updated through git on my RPI and tried again this morning, I had to a also edit linux/src/LinuxMotionTimer.cpp
for the ,'s at the end of list (2 of them).

Will try building also on Odroid and maybe grab on Edison and try it out there as well.

I also built the PS3 demo program so far. Note: in my current stuff, I have replace the ps3 code with my own joystick code that no longer requires you to use sudo to run...

Assuming this can all be accepted through a pull request to the master project, I will then merge my current joystick code in.

Wondering about some of the dependency stuff and how the install removes the built stuff... That is i I go into
src/hros1 and do a make. It builds everything. If I type make again, it again builds a lot of the stuff again (maybe all of it). I would think with the dependency stuff, it should not need to rebuild. But assuming that works, you then do a make install, which deletes most of the stuff, followed by a make clean which should finish cleaning stuff out...

Personally I wish that the projects could simply call off to make darwin (or hros1) library and if nothing changed, it would not need to do anything.

Also wondering if instead of building hros1.a we should be building something like libhros1.a or libdarwin.a and then potentially we could add -L../../lib to the link as well as -lhros1

But again I am just thinking out-loud.

Again thanks

KurtEck
08-18-2015, 09:17 AM
Quick update:
Builds on Odroid Xu3-lite - did not error out on ,s

Builds on Edison: dito about ,s. Ps3 demo did not link as I have real basic install on edison and did not fine -ljpeg... Probably something to install. I locally edited Makefile and removed the -ljpeg from linker line and then it built.

tician
08-18-2015, 09:44 AM
It was mostly a quick experiment in makefiles and automatically building in alternate directories (mostly from a way of compiling for multiple architectures (http://make.mad-scientist.net/papers/multi-architecture-builds/)), so not really planned well. Not entirely sure why it keeps rebuilding everything every single time, but it does; guessing it's not finding the dependencies correctly or I added/omitted a flag somewhere (or the build in alternate directory is rendering dependencies useless?). 'install' cleans the objects since it was rebuilding every time anyway, but can easily be modified to not cleanup objects afterwards. Installing in /usr/local/lib and /usr/local/include would be preferred once the archive is confirmed to be working correctly, then could use a '-ldarwinop' or '-lhros1' option with standard lib/include directories.

Brain incredibly borked at the moment from really long, really weird dreams last night.

Also, the reason minIni.cpp caused failure is because it is actually minIni.c.

edit: and I'm not even getting warnings about the extra comma in those enums...

tician
08-18-2015, 12:13 PM
Yep. It was how I was moving the object files around; they were not being found where expected, so always had to be rebuilt. Solution was to check for existence, then create if necessary, any directories required to have all the objects build in a duplicate directory tree of their source files instead of just one folder named 'obj'. Also means there can be multiple files with the same name and not overwrite each other, which is a big problem with the build everything in one folder method.

KurtEck
08-18-2015, 04:33 PM
Thanks, it is working better now.