A local copy of OpenSSL from GitHub
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

604 lines
24 KiB

  1. Design document for the unified scheme data
  2. ===========================================
  3. How are things connected?
  4. -------------------------
  5. The unified scheme takes all its data from the `build.info` files seen
  6. throughout the source tree. These files hold the minimum information
  7. needed to build end product files from diverse sources. See the
  8. section on `build.info` files below.
  9. From the information in `build.info` files, `Configure` builds up an
  10. information database as a hash table called `%unified_info`, which is
  11. stored in configdata.pm, found at the top of the build tree (which may
  12. or may not be the same as the source tree).
  13. [`Configurations/common.tmpl`](common.tmpl) uses the data from `%unified_info` to
  14. generate the rules for building end product files as well as
  15. intermediary files with the help of a few functions found in the
  16. build-file templates. See the section on build-file templates further
  17. down for more information.
  18. build.info files
  19. ----------------
  20. As mentioned earlier, `build.info` files are meant to hold the minimum
  21. information needed to build output files, and therefore only (with a
  22. few possible exceptions [1]) have information about end products (such
  23. as scripts, library files and programs) and source files (such as C
  24. files, C header files, assembler files, etc). Intermediate files such
  25. as object files are rarely directly referred to in `build.info` files (and
  26. when they are, it's always with the file name extension `.o`), they are
  27. inferred by `Configure`. By the same rule of minimalism, end product
  28. file name extensions (such as `.so`, `.a`, `.exe`, etc) are never mentioned
  29. in `build.info`. Their file name extensions will be inferred by the
  30. build-file templates, adapted for the platform they are meant for (see
  31. sections on `%unified_info` and build-file templates further down).
  32. The variables `PROGRAMS`, `LIBS`, `MODULES` and `SCRIPTS` are used to declare
  33. end products. There are variants for them with `_NO_INST` as suffix
  34. (`PROGRAM_NO_INST` etc) to specify end products that shouldn't get installed.
  35. The variables `SOURCE`, `DEPEND`, `INCLUDE` and `DEFINE` are indexed by a
  36. produced file, and their values are the source used to produce that
  37. particular produced file, extra dependencies, include directories
  38. needed, or C macros to be defined.
  39. All their values in all the `build.info` throughout the source tree are
  40. collected together and form a set of programs, libraries, modules and
  41. scripts to be produced, source files, dependencies, etc etc etc.
  42. Let's have a pretend example, a very limited contraption of OpenSSL,
  43. composed of the program `apps/openssl`, the libraries `libssl` and
  44. `libcrypto`, an module `engines/ossltest` and their sources and
  45. dependencies.
  46. # build.info
  47. LIBS=libcrypto libssl
  48. INCLUDE[libcrypto]=include
  49. INCLUDE[libssl]=include
  50. DEPEND[libssl]=libcrypto
  51. This is the top directory `build.info` file, and it tells us that two
  52. libraries are to be built, the include directory `include/` shall be
  53. used throughout when building anything that will end up in each
  54. library, and that the library `libssl` depend on the library
  55. `libcrypto` to function properly.
  56. # apps/build.info
  57. PROGRAMS=openssl
  58. SOURCE[openssl]=openssl.c
  59. INCLUDE[openssl]=.. ../include
  60. DEPEND[openssl]=../libssl
  61. This is the `build.info` file in `apps/`, one may notice that all file
  62. paths mentioned are relative to the directory the `build.info` file is
  63. located in. This one tells us that there's a program to be built
  64. called `apps/openss` (the file name extension will depend on the
  65. platform and is therefore not mentioned in the `build.info` file). It's
  66. built from one source file, `apps/openssl.c`, and building it requires
  67. the use of `.` and `include/` include directories (both are declared
  68. from the point of view of the `apps/` directory), and that the program
  69. depends on the library `libssl` to function properly.
  70. # crypto/build.info
  71. LIBS=../libcrypto
  72. SOURCE[../libcrypto]=aes.c evp.c cversion.c
  73. DEPEND[cversion.o]=buildinf.h
  74. GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
  75. DEPEND[buildinf.h]=../Makefile
  76. DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm
  77. This is the `build.info` file in `crypto/`, and it tells us a little more
  78. about what's needed to produce `libcrypto`. LIBS is used again to
  79. declare that `libcrypto` is to be produced. This declaration is
  80. really unnecessary as it's already mentioned in the top `build.info`
  81. file, but can make the info file easier to understand. This is to
  82. show that duplicate information isn't an issue.
  83. This `build.info` file informs us that `libcrypto` is built from a few
  84. source files, `crypto/aes.c`, `crypto/evp.c` and `crypto/cversion.c`.
  85. It also shows us that building the object file inferred from
  86. `crypto/cversion.c` depends on `crypto/buildinf.h`. Finally, it
  87. also shows the possibility to declare how some files are generated
  88. using some script, in this case a perl script, and how such scripts
  89. can be declared to depend on other files, in this case a perl module.
  90. Two things are worth an extra note:
  91. `DEPEND[cversion.o]` mentions an object file. DEPEND indexes is the
  92. only location where it's valid to mention them
  93. # ssl/build.info
  94. LIBS=../libssl
  95. SOURCE[../libssl]=tls.c
  96. This is the build.info file in `ssl/`, and it tells us that the
  97. library `libssl` is built from the source file `ssl/tls.c`.
  98. # engines/build.info
  99. MODULES=dasync
  100. SOURCE[dasync]=e_dasync.c
  101. DEPEND[dasync]=../libcrypto
  102. INCLUDE[dasync]=../include
  103. MODULES_NO_INST=ossltest
  104. SOURCE[ossltest]=e_ossltest.c
  105. DEPEND[ossltest]=../libcrypto.a
  106. INCLUDE[ossltest]=../include
  107. This is the `build.info` file in `engines/`, telling us that two modules
  108. called `engines/dasync` and `engines/ossltest` shall be built, that
  109. `dasync`'s source is `engines/e_dasync.c` and `ossltest`'s source is
  110. `engines/e_ossltest.c` and that the include directory `include/` may
  111. be used when building anything that will be part of these modules.
  112. Also, both modules depend on the library `libcrypto` to function
  113. properly. `ossltest` is explicitly linked with the static variant of
  114. the library `libcrypto`. Finally, only `dasync` is being installed, as
  115. `ossltest` is only for internal testing.
  116. When `Configure` digests these `build.info` files, the accumulated
  117. information comes down to this:
  118. LIBS=libcrypto libssl
  119. SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
  120. DEPEND[crypto/cversion.o]=crypto/buildinf.h
  121. INCLUDE[libcrypto]=include
  122. SOURCE[libssl]=ssl/tls.c
  123. INCLUDE[libssl]=include
  124. DEPEND[libssl]=libcrypto
  125. PROGRAMS=apps/openssl
  126. SOURCE[apps/openssl]=apps/openssl.c
  127. INCLUDE[apps/openssl]=. include
  128. DEPEND[apps/openssl]=libssl
  129. MODULES=engines/dasync
  130. SOURCE[engines/dasync]=engines/e_dasync.c
  131. DEPEND[engines/dasync]=libcrypto
  132. INCLUDE[engines/dasync]=include
  133. MODULES_NO_INST=engines/ossltest
  134. SOURCE[engines/ossltest]=engines/e_ossltest.c
  135. DEPEND[engines/ossltest]=libcrypto.a
  136. INCLUDE[engines/ossltest]=include
  137. GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
  138. DEPEND[crypto/buildinf.h]=Makefile
  139. DEPEND[util/mkbuildinf.pl]=util/Foo.pm
  140. A few notes worth mentioning:
  141. `LIBS` may be used to declare routine libraries only.
  142. `PROGRAMS` may be used to declare programs only.
  143. `MODULES` may be used to declare modules only.
  144. The indexes for `SOURCE` must only be end product files, such as
  145. libraries, programs or modules. The values of `SOURCE` variables must
  146. only be source files (possibly generated).
  147. `INCLUDE` and `DEPEND` shows a relationship between different files
  148. (usually produced files) or between files and directories, such as a
  149. program depending on a library, or between an object file and some
  150. extra source file.
  151. When `Configure` processes the `build.info` files, it will take it as
  152. truth without question, and will therefore perform very few checks.
  153. If the build tree is separate from the source tree, it will assume
  154. that all built files and up in the build directory and that all source
  155. files are to be found in the source tree, if they can be found there.
  156. `Configure` will assume that source files that can't be found in the
  157. source tree (such as `crypto/bildinf.h` in the example above) are
  158. generated and will be found in the build tree.
  159. The `%unified_info` database
  160. ----------------------------
  161. The information in all the `build.info` get digested by `Configure` and
  162. collected into the `%unified_info` database, divided into the following
  163. indexes:
  164. depends => a hash table containing 'file' => [ 'dependency' ... ]
  165. pairs. These are directly inferred from the DEPEND
  166. variables in build.info files.
  167. modules => a list of modules. These are directly inferred from
  168. the MODULES variable in build.info files.
  169. generate => a hash table containing 'file' => [ 'generator' ... ]
  170. pairs. These are directly inferred from the GENERATE
  171. variables in build.info files.
  172. includes => a hash table containing 'file' => [ 'include' ... ]
  173. pairs. These are directly inferred from the INCLUDE
  174. variables in build.info files.
  175. install => a hash table containing 'type' => [ 'file' ... ] pairs.
  176. The types are 'programs', 'libraries', 'modules' and
  177. 'scripts', and the array of files list the files of
  178. that type that should be installed.
  179. libraries => a list of libraries. These are directly inferred from
  180. the LIBS variable in build.info files.
  181. programs => a list of programs. These are directly inferred from
  182. the PROGRAMS variable in build.info files.
  183. scripts => a list of scripts. There are directly inferred from
  184. the SCRIPTS variable in build.info files.
  185. sources => a hash table containing 'file' => [ 'sourcefile' ... ]
  186. pairs. These are indirectly inferred from the SOURCE
  187. variables in build.info files. Object files are
  188. mentioned in this hash table, with source files from
  189. SOURCE variables, and AS source files for programs and
  190. libraries.
  191. shared_sources =>
  192. a hash table just like 'sources', but only as source
  193. files (object files) for building shared libraries.
  194. As an example, here is how the `build.info` files example from the
  195. section above would be digested into a `%unified_info` table:
  196. our %unified_info = (
  197. "depends" =>
  198. {
  199. "apps/openssl" =>
  200. [
  201. "libssl",
  202. ],
  203. "crypto/buildinf.h" =>
  204. [
  205. "Makefile",
  206. ],
  207. "crypto/cversion.o" =>
  208. [
  209. "crypto/buildinf.h",
  210. ],
  211. "engines/dasync" =>
  212. [
  213. "libcrypto",
  214. ],
  215. "engines/ossltest" =>
  216. [
  217. "libcrypto.a",
  218. ],
  219. "libssl" =>
  220. [
  221. "libcrypto",
  222. ],
  223. "util/mkbuildinf.pl" =>
  224. [
  225. "util/Foo.pm",
  226. ],
  227. },
  228. "modules" =>
  229. [
  230. "engines/dasync",
  231. "engines/ossltest",
  232. ],
  233. "generate" =>
  234. {
  235. "crypto/buildinf.h" =>
  236. [
  237. "util/mkbuildinf.pl",
  238. "\"\$(CC)",
  239. "\$(CFLAGS)\"",
  240. "\"$(PLATFORM)\"",
  241. ],
  242. },
  243. "includes" =>
  244. {
  245. "apps/openssl" =>
  246. [
  247. ".",
  248. "include",
  249. ],
  250. "engines/ossltest" =>
  251. [
  252. "include"
  253. ],
  254. "libcrypto" =>
  255. [
  256. "include",
  257. ],
  258. "libssl" =>
  259. [
  260. "include",
  261. ],
  262. "util/mkbuildinf.pl" =>
  263. [
  264. "util",
  265. ],
  266. }
  267. "install" =>
  268. {
  269. "modules" =>
  270. [
  271. "engines/dasync",
  272. ],
  273. "libraries" =>
  274. [
  275. "libcrypto",
  276. "libssl",
  277. ],
  278. "programs" =>
  279. [
  280. "apps/openssl",
  281. ],
  282. },
  283. "libraries" =>
  284. [
  285. "libcrypto",
  286. "libssl",
  287. ],
  288. "programs" =>
  289. [
  290. "apps/openssl",
  291. ],
  292. "sources" =>
  293. {
  294. "apps/openssl" =>
  295. [
  296. "apps/openssl.o",
  297. ],
  298. "apps/openssl.o" =>
  299. [
  300. "apps/openssl.c",
  301. ],
  302. "crypto/aes.o" =>
  303. [
  304. "crypto/aes.c",
  305. ],
  306. "crypto/cversion.o" =>
  307. [
  308. "crypto/cversion.c",
  309. ],
  310. "crypto/evp.o" =>
  311. [
  312. "crypto/evp.c",
  313. ],
  314. "engines/e_dasync.o" =>
  315. [
  316. "engines/e_dasync.c",
  317. ],
  318. "engines/dasync" =>
  319. [
  320. "engines/e_dasync.o",
  321. ],
  322. "engines/e_ossltest.o" =>
  323. [
  324. "engines/e_ossltest.c",
  325. ],
  326. "engines/ossltest" =>
  327. [
  328. "engines/e_ossltest.o",
  329. ],
  330. "libcrypto" =>
  331. [
  332. "crypto/aes.c",
  333. "crypto/cversion.c",
  334. "crypto/evp.c",
  335. ],
  336. "libssl" =>
  337. [
  338. "ssl/tls.c",
  339. ],
  340. "ssl/tls.o" =>
  341. [
  342. "ssl/tls.c",
  343. ],
  344. },
  345. );
  346. As can be seen, everything in `%unified_info` is fairly simple suggest
  347. of information. Still, it tells us that to build all programs, we
  348. must build `apps/openssl`, and to build the latter, we will need to
  349. build all its sources (`apps/openssl.o` in this case) and all the
  350. other things it depends on (such as `libssl`). All those dependencies
  351. need to be built as well, using the same logic, so to build `libssl`,
  352. we need to build `ssl/tls.o` as well as `libcrypto`, and to build the
  353. latter...
  354. Build-file templates
  355. --------------------
  356. Build-file templates are essentially build-files (such as `Makefile` on
  357. Unix) with perl code fragments mixed in. Those perl code fragment
  358. will generate all the configuration dependent data, including all the
  359. rules needed to build end product files and intermediary files alike.
  360. At a minimum, there must be a perl code fragment that defines a set of
  361. functions that are used to generates specific build-file rules, to
  362. build static libraries from object files, to build shared libraries
  363. from static libraries, to programs from object files and libraries,
  364. etc.
  365. generatesrc - function that produces build file lines to generate
  366. a source file from some input.
  367. It's called like this:
  368. generatesrc(src => "PATH/TO/tobegenerated",
  369. generator => [ "generatingfile", ... ]
  370. generator_incs => [ "INCL/PATH", ... ]
  371. generator_deps => [ "dep1", ... ]
  372. incs => [ "INCL/PATH", ... ],
  373. deps => [ "dep1", ... ],
  374. intent => one of "libs", "dso", "bin" );
  375. 'src' has the name of the file to be generated.
  376. 'generator' is the command or part of command to
  377. generate the file, of which the first item is
  378. expected to be the file to generate from.
  379. generatesrc() is expected to analyse and figure out
  380. exactly how to apply that file and how to capture
  381. the result. 'generator_incs' and 'generator_deps'
  382. are include directories and files that the generator
  383. file itself depends on. 'incs' and 'deps' are
  384. include directories and files that are used if $(CC)
  385. is used as an intermediary step when generating the
  386. end product (the file indicated by 'src'). 'intent'
  387. indicates what the generated file is going to be
  388. used for.
  389. src2obj - function that produces build file lines to build an
  390. object file from source files and associated data.
  391. It's called like this:
  392. src2obj(obj => "PATH/TO/objectfile",
  393. srcs => [ "PATH/TO/sourcefile", ... ],
  394. deps => [ "dep1", ... ],
  395. incs => [ "INCL/PATH", ... ]
  396. intent => one of "lib", "dso", "bin" );
  397. 'obj' has the intended object file with `.o`
  398. extension, src2obj() is expected to change it to
  399. something more suitable for the platform.
  400. 'srcs' has the list of source files to build the
  401. object file, with the first item being the source
  402. file that directly corresponds to the object file.
  403. 'deps' is a list of explicit dependencies. 'incs'
  404. is a list of include file directories. Finally,
  405. 'intent' indicates what this object file is going
  406. to be used for.
  407. obj2lib - function that produces build file lines to build a
  408. static library file ("libfoo.a" in Unix terms) from
  409. object files.
  410. called like this:
  411. obj2lib(lib => "PATH/TO/libfile",
  412. objs => [ "PATH/TO/objectfile", ... ]);
  413. 'lib' has the intended library file name *without*
  414. extension, obj2lib is expected to add that. 'objs'
  415. has the list of object files to build this library.
  416. libobj2shlib - backward compatibility function that's used the
  417. same way as obj2shlib (described next), and was
  418. expected to build the shared library from the
  419. corresponding static library when that was suitable.
  420. NOTE: building a shared library from a static
  421. library is now DEPRECATED, as they no longer share
  422. object files. Attempting to do this will fail.
  423. obj2shlib - function that produces build file lines to build a
  424. shareable object library file ("libfoo.so" in Unix
  425. terms) from the corresponding object files.
  426. called like this:
  427. obj2shlib(shlib => "PATH/TO/shlibfile",
  428. lib => "PATH/TO/libfile",
  429. objs => [ "PATH/TO/objectfile", ... ],
  430. deps => [ "PATH/TO/otherlibfile", ... ]);
  431. 'lib' has the base (static) library file name
  432. *without* extension. This is useful in case
  433. supporting files are needed (such as import
  434. libraries on Windows).
  435. 'shlib' has the corresponding shared library name
  436. *without* extension. 'deps' has the list of other
  437. libraries (also *without* extension) this library
  438. needs to be linked with. 'objs' has the list of
  439. object files to build this library.
  440. obj2dso - function that produces build file lines to build a
  441. dynamic shared object file from object files.
  442. called like this:
  443. obj2dso(lib => "PATH/TO/libfile",
  444. objs => [ "PATH/TO/objectfile", ... ],
  445. deps => [ "PATH/TO/otherlibfile",
  446. ... ]);
  447. This is almost the same as obj2shlib, but the
  448. intent is to build a shareable library that can be
  449. loaded in runtime (a "plugin"...).
  450. obj2bin - function that produces build file lines to build an
  451. executable file from object files.
  452. called like this:
  453. obj2bin(bin => "PATH/TO/binfile",
  454. objs => [ "PATH/TO/objectfile", ... ],
  455. deps => [ "PATH/TO/libfile", ... ]);
  456. 'bin' has the intended executable file name
  457. *without* extension, obj2bin is expected to add
  458. that. 'objs' has the list of object files to build
  459. this library. 'deps' has the list of library files
  460. (also *without* extension) that the programs needs
  461. to be linked with.
  462. in2script - function that produces build file lines to build a
  463. script file from some input.
  464. called like this:
  465. in2script(script => "PATH/TO/scriptfile",
  466. sources => [ "PATH/TO/infile", ... ]);
  467. 'script' has the intended script file name.
  468. 'sources' has the list of source files to build the
  469. resulting script from.
  470. Along with the build-file templates is the driving template
  471. [`Configurations/common.tmpl`](common.tmpl), which looks through all the
  472. information in `%unified_info` and generates all the rulesets to build libraries,
  473. programs and all intermediate files, using the rule generating
  474. functions defined in the build-file template.
  475. As an example with the smaller `build.info` set we've seen as an
  476. example, producing the rules to build `libcrypto` would result in the
  477. following calls:
  478. # Note: obj2shlib will only be called if shared libraries are
  479. # to be produced.
  480. # Note 2: obj2shlib must convert the '.o' extension to whatever
  481. # is suitable on the local platform.
  482. obj2shlib(shlib => "libcrypto",
  483. objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ],
  484. deps => [ ]);
  485. obj2lib(lib => "libcrypto"
  486. objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ]);
  487. src2obj(obj => "crypto/aes.o"
  488. srcs => [ "crypto/aes.c" ],
  489. deps => [ ],
  490. incs => [ "include" ],
  491. intent => "lib");
  492. src2obj(obj => "crypto/evp.o"
  493. srcs => [ "crypto/evp.c" ],
  494. deps => [ ],
  495. incs => [ "include" ],
  496. intent => "lib");
  497. src2obj(obj => "crypto/cversion.o"
  498. srcs => [ "crypto/cversion.c" ],
  499. deps => [ "crypto/buildinf.h" ],
  500. incs => [ "include" ],
  501. intent => "lib");
  502. generatesrc(src => "crypto/buildinf.h",
  503. generator => [ "util/mkbuildinf.pl", "\"$(CC)",
  504. "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
  505. generator_incs => [ "util" ],
  506. generator_deps => [ "util/Foo.pm" ],
  507. incs => [ ],
  508. deps => [ ],
  509. intent => "lib");
  510. The returned strings from all those calls are then concatenated
  511. together and written to the resulting build-file.