slitaz-doc-wiki-data view pages/en/devnotes/new-tazwok-illustrated.txt @ rev 7
Add pages/en folder.
| author | Christopher Rogers <slaxemulator@gmail.com> |
|---|---|
| date | Sat Feb 26 12:17:18 2011 +0000 (2011-02-26) |
| parents | |
| children | df9950c99d41 |
line source
1 ====== The new tazwok illustrated! ======
3 Some items in revenue no longer needed with the new version of tazwok. During migration, a number of problems appear.The information on these two points are available below, with examples.
5 By simplifying the writing of recipes, we simplify the work and contribution that benefits everyone!
7 ==== DEPENDS/BUILD_DEPENDS : ====
8 Tazwok now uses the dependent variable for finding the necessary compilation dependencies.
9 Here's how it works:
11 * Tazwok tree list of all dependencies from the dependent variable
12 * For each package, if there is a shareholder *-dev package, it adds it to the dependencies
13 * Tazwok done the same for BUILD_DEPENDS.
15 So far, when a packet was both dependency and build dependency, the recipe looked like this:
17 <file>
18 DEPENDS="pkgX"
19 BUILD_DEPENDS="pkgX pkgX-dev"
20 </file>
21 Now that is enough:
22 <file>
23 DEPENDS="pkgX"
24 </file>
25 Where there were trees more complex dependencies, the recipe looked like this:
26 <file>
27 # pkgY depend's on pkgX
28 DEPENDS="pkgY"
29 BUILD_DEPENDS="pkgY pkgY-dev pkgX pkgX-dev"
30 </file>
31 Now that is enough:
32 <file>
33 DEPENDS="pkgY"
34 </file>
36 The recipes also contain many redundancies in the definition of addiction, for example:
37 <file>
38 # pkgY depend's on pkgX
39 DEPENDS="pkgY pkgX"
40 </file>
41 Here, needless to say since pkgX will be installed along pkgY anyway (Tazpkg manages dependencies automatically!).
43 By following these three councils, it appears that about half the packages in DEPENDS / BUILD_DEPENDS can be removed without changing the revenue system behavior, it is not nothing!
45 <note tip>An automated cleaning using some scripts is expected, after all recipes have been compiled at least once successfully using the new version of tazwok; In the meantime, these tips can be applied to writing new recipe for simplicity or manually with existing revenues when updating / corrections.</note>
47 **Examples:**
48 * graveman: http://hg.slitaz.org/wok/rev/7f0604e0bde0
49 * enlightenment & cie: http://hg.slitaz.org/wok/rev/85cd798d6997
51 ==== TARBALL/WGET_URL/SOURCE/download from the VCS ====
53 This is important: always put the necessary tools to download / decompression sources in DEPENDS or BUILD_DEPENDS. This allows to define tazwok right firing order (do not try to cook a package that needs wget wget before himself).
55 //The packets affected by this://
56 * wget url for https, ftps and some URLs that busybox does not include
57 * mercurial/subversion/git: they are used to obtain the source
58 * tar/unzip: sometimes necessary to unpack the sources.
60 By default, tazwok re-sources in compact format .tar.lzma. He calls PACKAGE-VERSION.tar.lzma or SOURCE-VERSION.tar.lzma if SOURCE is defined. Note: Choose the name of the archive is now the only function of the variable SOURCE!
62 Tazwok now supports files or "weird" URL (download.php?version=foo&blah=Idontknowwhat). The logic is: if WGET_URL does not end with tarball, then names the file downloaded tarball.
64 Tazwok also supports the use of mercurial/subversion/git in WGET_URL. The syntax is:
65 <file>WGET_URL="subversion|svn://svn.mplayerhq.hu/mplayer/trunk</file>
66 An optional variable is BRANCH: it allows to specify the revision /tag/branch to use (see examples below). Where BRANCH is used, it is important that $VERSION is part of its definition.
68 Note that the sources will be obtained through the requested tool, then packaged in .tar.lzma. The archive will be named as explained above. This means that the variable source can be used to ensure that many recipes use the same repository without creating multiple archives.
70 First, it helps to know what revision is installed when using the package manager. Second, it allows to differentiate tazwok compressed sources. Indeed, if the archive keeps the same name, it will not be re-downloaded, which is undesirable when trying to update the package.
72 **Examples:**
73 * Here was necessary wget: http://hg.slitaz.org/wok/rev/012847ddd0cb
74 * Tinyproxy did not report the URL of its source code is corrected: http://hg.slitaz.org/wok/rev/25967da0e1af
75 * WGET_URL now supports xpi: http://hg.slitaz.org/wok/rev/37738b3ee08f
76 * WGET_URL with a "weird" URL: http://hg.slitaz.org/wok/rev/102de15fea8d
77 * WGET_URL using git: http://hg.slitaz.org/wok/rev/e06d60ae03eb
78 * WGET_URL using subversion: http://hg.slitaz.org/wok/rev/c4c54646489a
79 * WGET_URL using mercurial: http://hg.slitaz.org/wok/rev/756ed4b1daac
80 * It was difficult to choose how to define BRANCH and VERSION for aufs: http://hg.slitaz.org/wok/rev/67231cfc5475
81 * Here are two sources of records were in conflict, resolved by SOURCE: http://hg.slitaz.org/wok/rev/b891cba4f48e
82 * slitaz-dev-tools contains the sources for SliTaz tools that contain very little code, using SOURCE="slitaz-dev-tools" in recipes that use this deposit to avoid having duplicate tarballs: http://hg.slitaz.org/wok/rev/808826645cc2
84 ==== Exceptions dependencies cooking ====
85 In some cases, no dependence cooking is installed:
86 * For recipes with WANTED
87 * For recipes without compile_rules()
89 Note that packets may be required to obtain / decompressing the source code will still be installed if they are in DEPENDS / BUILD_DEPENDS. These are wget, mercurial, subversion, git, tar and unzip.
91 If you do not compile_rules() but want to force the installation of all dependencies of cooking, there is a little hack:
92 <file>
93 compiles_rules()
94 {
95 :
96 }
97 </file>
99 **Examples:**
100 * Removal of compiles_rules() to avoid installing unnecessary dependencies Cooking: http://hg.slitaz.org/wok/rev/f579356b437f
101 * Remove a hack with fake compiles_rules was useless ... http://hg.slitaz.org/wok/rev/5b4581f8e476
103 ==== Define src/_pkg & move in the right place (hacks in the recipe) ====
105 By default, the new sources in place tazwok $WOK/$PACKAGES/$PACKAGE-$VERSION: it renames the parent directory of sources if necessary. So far, $src was not properly defined for recipes using both SOURCE and WANTED. Many recipes implement their own solution in different ways, which is difficult to consider a standardized way and can cause compatibility problems.
107 If tazwok detects src=/_pkg= in a recipe, it continues to use the old behavior to ensure compatibility (this produces errors in some cases). It is no longer necessary and not ideal.
109 The hacks in the recipe source that move in the right place are no longer needed either, and can also cause problems.
111 In conclusion, it is better to consider that $src/$_pkg are defined by default and try to rely as much as possible.
113 ** Examples: **
114 * Removing src= by Godane: http://hg.slitaz.org/wok/rev/a1c1d35d9f92
115 * src=/_pkg= can / should also be removed from WANTED: http://hg.slitaz.org/wok/rev/07adb7cbd0c8
116 * Here, an old hack was the problem: http://hg.slitaz.org/wok/rev/62f6142d9fb3
117 * Sources are now //always// placed in a sub-directory $src http://hg.slitaz.org/wok/rev/e64069568fe7
118 * Another case: call the configure script from a folder separate compilation (*-build): http://hg.slitaz.org/wok/rev/7461a0c31d62
119 * Fixed dmraid: http://hg.slitaz.org/wok/rev/f5b7e0c47763 http://hg.slitaz.org/wok/rev/59ea9409ad8a
121 ==== Set the default paths in configure: ====
122 <note tip>See /etc/slitaz/slitaz.conf, /etc/config.site and the new revenue model in place by New tazwok tree</note>
124 The new version of tazwok attempts to pass the default paths to configure using the environment variable CONFIG_SITE calling /etc/config.site, which works in most cases. Nevertheless configure scripts are specific to each source and sometimes CONFIG_SITE not be supported. For this reason, the best way to remove the definitions of paths is unnecessary to do so on a case by case, when the updated recipe, and make sure everything works.
126 In rare cases, this new product functionality problems. It happens that some recipes that did not use the default paths used by CONFIG_SITE now, and an update function genpkg_rules() is then mandatory.
128 ** Examples: **
129 * A file or did not install properly in acl is corrected by CONFIG_SITE: http://hg.slitaz.org/wok/rev/f831ecb652a6
130 * Another example: http://hg.slitaz.org/wok/rev/259214792e30
132 <note tip>CONFIG_SITE= can be used in recipes to use a different file than the default (can be useful for packages gnome or something like that ...)</note>
134 ==== DESTDIR=$PWD/_pkg ====
136 DESTDIR is passed to make install using the environment variable of the same name. The new path for installation is $ WOK/$PACKAGE/install. This will remove the source folder after packaging, it does not contain any file used by a recipe in his genpkg_rules().
138 Most recipes still use DESTDIR=$PWD/_pkg. However, if no revenue is redefining the variables src/_pkg, automatically move to the tazwok $WOK/$PACKAGE/install.
140 In some cases, as with other variables, DESTDIR is not taken into account or the package is not installed by make. In these cases, the variable $DESTDIR is available to define the installation directory in the recipe.
142 In rare cases, this behavior causes incompatibilities. This happens when revenues define the path to the installation folder without using src/_pkg. The solution is not to set these paths in revenue (that calling the main recipe with WANTED included), make sure the installation is done well in $WOK/$PACKAGE/install and trust the variables provided by tazwok.
144 ** Examples: **
145 * Remove _pkg= & DESTDIR= same time for this to work: http://hg.slitaz.org/wok/rev/cf088243a4a5
146 * Remove references "useless" to $src to the sources are withdrawn: http://hg.slitaz.org/wok/rev/0731792c3994 http://hg.slitaz.org/wok/rev/5d6340961543
147 * Bash does not take into account DESTDIR environment variable: http://hg.slitaz.org/wok/rev/fa7b7514e1d8
148 * acl attr does not include DESTDIR (in this installation the destination was still $PWD/_pkg): http://hg.slitaz.org/wok/rev/fa7b7514e1d8
150 ==== MAKEFLAGS ====
152 MAKEFLAGS also increased to make using environment variables, and again this does not always work. In most cases, -j4 can be removed. In some cases it is necessary to use MAKEFLAGS directly to make the recipe:
153 make $MAKEFLAGS
155 Tazwok automatically sets the value for $MAKEFLAGS in the number of heart that contains the processor -j4 should be removed from all the recipes can be compiled on computers that have more resources (4 cores can use -j5)
157 **Problems MAKEFLAGS:**
159 So far, only recipes with-j4 compiled using the multi-threaded, whereas now all make and make install use. This behavior can cause errors. Some sources do not support multi-threaded compilation but do not disable. This is the most common problem associated with the changes explained here.
161 //Problem in compiling://
163 During compilation, it happens that rely on other libraries compiled with the same sources. If they are compiled at the same time, this causes an error about a missing library. In this case, we see in the compilation text that the library in question has begun to be built few lines earlier, but that this process was not finished yet. To resolve this problem, add -j1 to make. It is the most common mistake, there are other more rare that take a similar form.
165 //Problem in installation://
167 The characteristic of this error is that the installation stops and an error message says that it is impossible to create a folder because it already exists: a parallel process is actually creating it. In this case, add-j1 to make install.
169 ** Examples: **
170 * Several changes explained here in the recipe for gettext: http://hg.slitaz.org/wok/rev/9411655af0e2
172 ==== Variables $stuff, $wanted_stuff and $fs ====
174 Now the variable $stuff is available and returns the record stuff in the recipe, it uses an absolute path. The variable $wanted_stuff refers to the file defined in the package stuff WANTED, if any. The variable $fs refers to the future content of the package in taz/*/fs, as before, the difference is that now $fs uses an absolute path
176 ** Examples: **
177 * A commit with several changes regarding the variable $stuff: http://hg.slitaz.org/wok/rev/be13f25e790b
178 * A correction necessary when we have made an absolute path $fs: http://hg.slitaz.org/wok/rev/8c897d2542ab
180 ==== Do not use 'exit' but 'return' ====
182 Now when cooking several packages with a list tazwok does not call for new tazwok cook. There is only one session tazwok so that the execution is faster. If a recipe uses exit, it leaves tazwok and following list is not cooked.
184 **Example:**
185 * Removing all exit the wok recipes: http://hg.slitaz.org/wok/rev/0b4cf0d9e1b5
187 ==== Conclusion - What to do when updating a recipe: ====
189 * Remove src=/_pkg= recipe and those who declare as WANTED.
190 * Remove DESTDIR=$PWD/_pkg and if it does not work, or if the means to define the installation directory is not + make DESTDIR, rather than using $ DESTDIR $PWD/_pkg.
191 * Remove the definition of default paths and see if it works, otherwise leave.
192 * Remove-j4 and see if it works; If the mutli-threaded does not work, re-activate using $MAKEFLAGS; if multi-threading causes problems, add-j1 to the right place.
193 * Remove BUILD_DEPENDS / DEPENDS redundant.
194 * Check that packages are created correctly, otherwise update the paths in genpkg_rules().
195 * Try to declare all sources of revenue in that SliTaz can be compiled without internet connection (requires to download any source before).
196 * Check that the package needed to download / extract the source code are defined in BUILD_DEPENDS.
197 * Check as EXIT is not used in the recipe.
199 ==== Some more complex cases ... ====
201 I put them there in the end because there are already too many to be integrated:)
202 The items below correspond to specific cases.
204 == Variable COOK_OPT ==
206 This new variable can contain options that alter the behavior of tazwok. They are useful in very special cases.
208 **genpkg=**
210 In the recipe PACKAGE defines a priority order for the revenue package containing WANTED="PACKAGES" (and only them!). If you include multiple packages, separate them with double points ':'. If packets are not defined in this option, they will be packaged later, in alphabetical order (default)
212 Used in glibc: http://hg.slitaz.org/wok/file/tip/glibc/receipt
214 **!repack_src**
216 Disable re-compression format sources .tar.lzma.
218 Ruby-pkgconfig used for the sources remain in gem: http://hg.slitaz.org/wok/file/tip/ruby-pkgconfig/receipt
220 **!unpack**
222 Prevents decompression of the archive-source in the wok.
224 This is used by ruby-pkgconfig as well (see link above)
226 This is the only case yet!
228 ==== Cooking the toolchain ====
230 To cook the toolchain SliTaz, we use a temporary toolchain. Some recipes use specific rules in this step. When cooking this toolchain temporarily affected software packaged but are not directly installed in the chroot purpose built. The affected packages are listed in the variable SLITAZ_TOOLCHAIN configuration file /etc/slitaz/slitaz.conf
232 Additional features are:
233 * Precook_tmp_toolchain() - Used only by gcc & binutils for the moment, because they are cooked twice during the preparation of the temporary toolchain.
234 * Cook_tmp_toolchain() - Used by the most packets SLITAZ_TOOLCHAIN to define how they should be compiled for the temporary toolchain. When cook_tmp_toolchain() is absent, compile_rules() is used instead. This avoids writing two identical functions. Note that in this case. ./configure does not set the default paths in the recipe, because the temporary toolchain must be able to do it via the environment variable CONFIG_SITE. Indeed, packets compiled during this stage are not installed in the usual place but in /tools.
236 ** Examples: **
237 * Binutils: http://hg.slitaz.org/wok/file/tip/binutils/receipt
238 * Gettext: http://hg.slitaz.org/wok/file/tip/gettext/receipt
239 * Bash: http://hg.slitaz.org/wok/file/tip/bash/receipt
240 * Patch does not need cook_tmp_toolchain(): http://hg.slitaz.org/wok/file/tip/patch/receipt
241 * Autoconf either: http://hg.slitaz.org/wok/file/tip/autoconf/receipt
243 == tazwok get-src / report in recipes ==
245 Report is a module libtaz for organizing the display commands in the terminal and create logs available including http://bb.slitaz.org interface. It can be used in recipes, as follows (that is abstract, the following examples of actual application):
247 <file>
248 compile_rules() # Per example
249 {
250 report open-bloc #compiles_rules is a step, declaring that there will substeps
251 report step "Action machin"
252 ...
253 report step "Action truc"
254 ..
255 report close-bloc #Close the open block previously
256 }
257 </file>
259 Specifically, there is one case where we use it: when using get-src tazwok PACKAGE --target=... . This command creates a new step (postponement step). We need to open a block before and after the close, as well as adding several other deferral step "..." to log and display in the terminal is correct. Each farm deferral step the previous step, if we do not open the block, get-src shut tazwok step "Executing compiles rules"
261 Delaying closed block must be completely executed, otherwise the log/display will be broken. That's why we use { report-block closed; return 1; } return rather than alone.
263 The practical use of this tazwok get-src is that you can unpack the sources of designated PACKAGE at target.
265 In the examples below, observe the correlation between the delay step and displayed in the log. Observe also the correlation between tazwok get-src and the message "Checking for source tarball ..." in the log. You learn how to carry report open-bloc/closed block create a subset in genpkg_rules (named "Executing compile_rules" in the log). If it were not for open-bloc/close-bloc, the new steps would be posted to the result of "Executing compile_rules" This is not what we wanted.
267 ** Examples (recipe + log): **
268 * Linux needs patches contained in the sources of aufs, Godane took the opportunity to improve the log. Recipe: http://hg.slitaz.org/wok/file/tip/linux/receipt; log: http://bb.slitaz.org/log.php?version=cooking&package=linux
269 * Gcc uses several other sources of packages during the cooking of the temporary toolchain. Recipe: http://hg.slitaz.org/wok/file/tip/gcc/receipt; log: http://bb.slitaz.org/log.php?version=cooking&package=tmp-toolchain-gcc
270 * mingw32-gcc was corrected using this approach, it also allowed to declare all sources used. Commit: http://hg.slitaz.org/wok/rev/fd43246b4613; log: http://bb.slitaz.org/log.php?version=cooking&package=mingw32-gcc