p≡p JSON adapter
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

535 lines
20 KiB

5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
  1. # p≡p JSON Server Adapter
  2. ## Introduction
  3. The p≡p JSON Server Adapter provides a REST-like jQuery-compatible API to
  4. connect with the p≡p engine. It is language-independent and can be used by
  5. any client.
  6. ## Requirements
  7. In order to use the p≡p JSON Server Adapter, you need to build and run it.
  8. Currently, Linux (Debian 9, Ubuntu 16.04) and MacOS (10.11, 10.12) are
  9. supported, Windows is about to follow. Newer versions should also work
  10. (file a bug report if not) but are not in our main focus, yet.
  11. ## Dependencies
  12. * C++ compiler: tested with g++ 4.8 and 4.9, and clang++ 2.8. Newer versions should work, too.
  13. * GNU make
  14. * libboost-thread-dev (tested with 1.58)
  15. * libboost-program-options-dev (tested with 1.58)
  16. * libboost-filesystem-dev (tested with 1.58)
  17. * libevent-dev 2.0.21 or 2.0.22 (or build from source, see below)
  18. * [p≡p Engine](https://letsencrypt.pep.foundation/trac/wiki/Basic%20Concepts%20of%20p%E2%89%A1p%20engine)
  19. (which needs gpgme-thread, a patched libetpan, libboost-system-dev)
  20. * OSSP libuuid
  21. ## Building/Installing
  22. ### Install the dependencies
  23. Debian 9:
  24. ~~~~~
  25. apt install -y build-essential libboost1.62-dev libboost-system1.62-dev \
  26. libboost-filesystem1.62-dev libboost-program-options1.62-dev \
  27. libboost-thread1.62-dev libgpgme-dev uuid-dev googletest
  28. ~~~~~
  29. macOS 10.12:
  30. Use homebrew or macports to install the required libraries.
  31. For more explicit instructions on how to do this with macports, see the
  32. section below.
  33. Build and install the pEp Engine. Instructions can be found here:
  34. [https://cacert.pep.foundation/dev/repos/pEpEngine/file/ef23982e4744/README.md](https://cacert.pep.foundation/dev/repos/pEpEngine/file/ef23982e4744/README.md).
  35. ### Build and install libevent
  36. ~~~~~
  37. mkdir ~/code/json-ad
  38. hg clone https://cacert.pep.foundation/dev/repos/pEpJSONServerAdapter/ ~/code/json-ad
  39. cd ~/code/json-ad/libevent-2.0.22-stable
  40. ./configure --prefix="$HOME/code/json-ad/libevent-2.0.22-stable/build/" --disable-openssl
  41. make
  42. make install
  43. ~~~~~
  44. ### Build and install the JSON server
  45. ~~~~~
  46. cd ~/code/json-ad/server
  47. ~~~~~
  48. Edit the build configuration to your needs in `./Makefile.conf`, or create a
  49. `./local.conf` that sets any of the make variables documented in
  50. `./Makefile.conf`.
  51. If a dependency is not found in your system's default include or library
  52. paths, you will have to specify the according paths in a make variable.
  53. Typically, this has to be done at least for the pEp Engine, libetpan and
  54. libevent.
  55. Below are two sample `./local.conf` files, for orientation.
  56. macOS 10.12:
  57. ~~~~~
  58. PREFIX=$(HOME)/code/json-ad/build
  59. HTML_DIRECTORY=$(PREFIX)/share/pEp/json-adapter/html
  60. GTEST_DIR=$(HOME)/code/gtest/googletest
  61. BOOST_INC=-I$(HOME)/Cellar/boost/1.65.1/include
  62. BOOST_LIB=-L$(HOME)/Cellar/boost/1.65.1/lib
  63. ENGINE_INC=-I$(HOME)/code/engine/build/include
  64. ENGINE_LIB=-L$(HOME)/code/engine/build/lib
  65. ETPAN_INC=-I$(HOME)/code/libetpan/build/include
  66. ETPAN_LIB=-L$(HOME)/code/libetpan/build/lib
  67. EVENT_INC=-I$(HOME)/code/json-ad/libevent-2.0.22-stable/build/include
  68. EVENT_LIB=-L$(HOME)/code/json-ad/libevent-2.0.22-stable/build/lib
  69. GPGME_INC=-I$(HOME)/Cellar/gpgme/1.9.0_1/include
  70. GPGME_LIB=-L$(HOME)/Cellar/gpgme/1.9.0_1/lib
  71. UUID_INC=-I$(HOME)/Cellar/ossp-uuid/1.6.2_2/include
  72. UUID_LIB=-L$(HOME)/Cellar/ossp-uuid/1.6.2_2/lib
  73. ~~~~~
  74. Debian 9:
  75. ~~~~~
  76. PREFIX=$(HOME)/code/json-ad/build
  77. HTML_DIRECTORY=$(PREFIX)/share/pEp/json-adapter/html
  78. GTEST_DIR=/usr/src/googletest/googletest/
  79. ENGINE_INC=-I$(HOME)/code/engine/build/include
  80. ENGINE_LIB=-L$(HOME)/code/engine/build/lib
  81. ETPAN_INC=-I$(HOME)/code/libetpan/build/include
  82. ETPAN_LIB=-L$(HOME)/code/libetpan/build/lib
  83. EVENT_INC=-I$(HOME)/code/json-ad/libevent-2.0.22-stable/build/include
  84. EVENT_LIB=-L$(HOME)/code/json-ad/libevent-2.0.22-stable/build/lib
  85. ~~~~~
  86. Now, build and install the server:
  87. ~~~~~
  88. make all
  89. make install
  90. ~~~~~
  91. With `make test` you can execute the server's tests.
  92. ### Macports
  93. [Install MacPorts](https://www.macports.org/install.php) for your version of macOS.
  94. If MacPorts is already installed on your machine, but was installed by a
  95. different user, make sure your `PATH` variable is set as follows in
  96. `~/.profile`:
  97. ```
  98. export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
  99. ```
  100. Install dependencies packaged with MacPorts as follows.
  101. ```
  102. sudo port install gpgme boost ossp-uuid
  103. ```
  104. ## Running the pEp JSON Adapter
  105. You can use `make run` to start the server.
  106. 1. Run ./pep-json-server. This creates a file that is readable only by the
  107. current user (~/.pEp/json-token-${USER}) and contains the address and
  108. port the JSON adapter is listening on, normally 127.0.0.1:4223 and a
  109. "security-token" that must be given in each function call to authenticate
  110. you as the valid user.
  111. ```
  112. ./pep-json-server
  113. ```
  114. 2. Visit that address (normally http://127.0.0.1:4223/) in your
  115. JavaScript-enabled web browser to see the "JavaScript test client".
  116. 3. Call any function ("version()" or "get_gpg_path()" should work just
  117. fine) with the correct security token.
  118. ## Using the p≡p JSON Adapter
  119. In the following section, you'll find background information on how to use
  120. the adapter and its functions.
  121. ### Server startup and shutdown
  122. The JSON Server Adapter can be started on demand.
  123. It checks automatically whether an instance for the same user on the machine
  124. is already running and if yes it ends itself gracefully.
  125. If there is no running server found the newly started server creates the
  126. server token file and forks itself into background (if not prevented via
  127. "-d" commandline switch).
  128. ### Session handling
  129. When using the p≡p engine, a session is needed to which any adapter can
  130. connect. The p≡p JSON Server Adapter automatically creates one session per
  131. HTTP client connection (and also closes that session automatically when the
  132. client connections is closed). Therefore, the client does not need to take
  133. care of the session management. However, the client has to set up a [HTTP
  134. persistent
  135. connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection).
  136. ### API Principles
  137. All C data types are mapped the same way, so some day the JSON wrapper can
  138. be generated from the p≡p Engine header files (or the JSON wrapper and the
  139. p≡p engine header are both generated from a common interface description
  140. file).
  141. | C type | JSON mapping |
  142. |--|--|
  143. | `bool` | JSON boolean |
  144. | `int` | JSON decimal number |
  145. | `size_t` | JSON decimal number |
  146. | `char*` (representing a UTF-8-encoded NULL-terminated string | JSON string |
  147. | `char*` (representing a binary string | base64-encoded JSON string |
  148. | `enum` | either JSON decimal number or JSON object containing one decimal number as member |
  149. | `struct` | JSON object |
  150. | linked lists (e.g. `bloblist_t`, `stringlist_t`, `identity_list` etc.) | JSON array of their member data type (without the `next` pointer) |
  151. The parameter type PEP_SESSION is handled automatically by the JSON Server
  152. Adapter and the PEP_SESSION parameter is omitted from the JSON API.
  153. #### enum types
  154. Enum types are represented as JSON objects with one member, whose name is
  155. derived from the enum type name, holding the numeric value of the enum.
  156. Some enum types are still represented directly as JSON decimal number. It
  157. shall be changed in a future version of the JSON Adapter.
  158. #### String types
  159. The JSON Server Adapter does automatic memory management for string
  160. parameters. The current p≡p Engine's API distinguish between `const char*`
  161. parameters and `char*` parameters. `const char*` normally means: the
  162. "ownership" of the string remains at the caller, so the JSON Adapter frees
  163. the string automatically after the call. `char*` normally means: the
  164. "ownership" of the string goes to the Engine, so the JSON Adapter does _not_
  165. free string.
  166. If there are functions that have a different semantics the behavior of the
  167. JSON wrapper has to be changed.
  168. #### Parameter (value) restrictions
  169. Some API functions have restrictions on their parameter values. The JSON
  170. Adapter does not know these restrictions (because it does not know the
  171. semantics of the wrapped functions at all). So it is the client's
  172. responsibility to fulfill these parameter restrictions! Especially when
  173. there are restrictions that are checked with assert() within the p≡p Engine,
  174. it is impossible for the JSON Adapter to catch failed assertions - the
  175. Engine and the Adapter process will be terminated immediatetely when the
  176. Engine is compiled in debug mode (= with enabled assert() checking).
  177. Currently there are no range checks for numerical parameter types (e.g. a
  178. JSON decimal number can hold a bigger value than the `int` parameter type of
  179. a certain C function).
  180. ### API Reference
  181. An complete overview with all functions that are callable from the client
  182. can be found in the [API Reference](pEp JSON Server Adapter/API Reference).
  183. That API reference is a generated file that shows the current API briefly.
  184. There is also a (currently manually written) file that holts a copy of the
  185. documentation from the Engine's header files: [API reference detail.md]
  186. Most of the callable functions are functions from the C API of the p≡p
  187. Engine. They are described in detail, incl. pre- and post-conditions in
  188. the appropriate C header files of the Engine.
  189. ### Authentication
  190. The JSON Server Adapter and the client have to authenticate to each other.
  191. "Authentication" in this case means "run with the same user rights". This is
  192. done by proving that each communication partner is able to read a certain
  193. file that has user-only read permissions.
  194. 0. There is a common (between client & server) algorithm to create the path
  195. and filename of the "server token file", for a given user name.
  196. The token file and its directory MUST be owned by the user and MUST be
  197. readable and writable only by the user, nobody else. Client and server
  198. check for the right ownership and access rights of the token file and its
  199. directory. (TODO: What shall be done if that check fails?)
  200. 1. The server creates a "server token file" containing a "server token" (a
  201. random-generated string of printable ASCII characters) and the IP address
  202. and port where the server listens on. This file can only be read by
  203. client programs that run with the same user rights.
  204. 2. The client checks the path, reads the "server token" from the file and
  205. authenticates itself to the server in each JSON RPC call with that "server
  206. token".
  207. ## Extending / customizing
  208. If you want to extend or customize the p≡p JSON Adapter, there are several
  209. rules and definitions to take into account.
  210. ### Definitions
  211. * The `FunctionMap function` in `ev_server.cc` defines which functions
  212. are callable via the JSON-RPC interface. The existing entries show the
  213. syntax of that map.
  214. Non-static member functions can be called, too. Thanks to `std::function<>`
  215. a member function `Foo::func(Params...)` is handled like a free-standing
  216. function `func(Foo* f, Params...)`.
  217. * For each type there must exist specializations of the template classes
  218. "In" (for input parameters) and "Out" (for output parameters).
  219. The linker will tell you, which specializations are needed.
  220. * The specializations for "generic types" are in `function_map.cc`.
  221. * The specializations for "p≡p-specific types" are in `pep-types.cc`.
  222. #### Parameter directions (In, Out, InOut)
  223. The p≡p JSON Server Adapter supports Input, Output and two ways of "In/Out"
  224. parameters. You have to annotate the direction in the FunctionMap with
  225. `In<>` for input, `Out<>` for output and InOut<> or InOutP<> for in/out
  226. parameters. These wrapper classes have an optional second template
  227. parameter (parameter type flag) that is explained below.
  228. Return values are always "output" parameters, so they don't have to be
  229. wrapped with `Out<>`, but this wrapper is necessary when you need
  230. non-default wrapper semantics, see below.
  231. Input parameters of fundamental or simple struct types are
  232. usually by-value parameters. Complex structs (or structs that are only
  233. forward-declared in the public API) are usually pointer
  234. parameters. Both ways are supported. You have to specialize `In<T>` or
  235. `In<T*>`, depending how your type is used.
  236. Output parameters of fundamental or simple struct types `T` are usually
  237. declared as a paremeter of type `T*`. The p≡p JSON Server Adapter manages
  238. the memory allocated by the called C function automatically and calls the
  239. appropriate de-allocating function after use.
  240. Calling a function with output parameters requires a dummy value (`null` or
  241. empty string is fine) at the JSON side for each output parameter to keep the
  242. number of parameters at the JSON side the same with the C side.
  243. For In/Out parameters there exist two calling conventions for
  244. call-by-pointer types:
  245. 1. caller allocates object and fills with input values, callee can only change members.
  246. The C type of the parameter is usually `struct T*`. Use the wrapper `InOut<>`
  247. for these parameters.
  248. 2. caller allocates object and fills with input values, callee might
  249. change/reallocate the whole object. The C type of the parameter is
  250. `struct T**`. Use the wrapper `InOutP<>` in these cases.
  251. `InOutP<>` is also the right wrapper for in/out parameters of fundamental or
  252. enum types due to the additional indirection in the C function call
  253. signature.
  254. #### Parameter type flags
  255. The wrapper classes might be instantiated with special "parameter type
  256. flags". If no flag is given the `DefaultFlag` is used with means the
  257. semantics described already above.
  258. At the moment there exist two parameter type flags which are interpreted as
  259. bitfield, so they can be combined:
  260. * NoInput : This flags a parameter at the C side that shall not be exposed
  261. at the JSON side. So the value cannot be specified by the client, it is
  262. provided by the JSON Server Adapter internally (e.g. for PEP_SESSION)
  263. * DontOwn : Used for pointer types who don't "own" the referred ressource,
  264. so it is not released automatically by the JSON Server Adapter after the
  265. call.
  266. More flags will be added when different semantics will be needed.
  267. ### Automatic parameter value generation
  268. For some parameters or parameter combinations the JSON Server Adapter is
  269. able to generate the values automatically either from the environment or
  270. from other parameters.
  271. These automatic parameter value generators are supported at the moment:
  272. #### InLength
  273. For functions that have a string parameter of type `const char*` followed by
  274. a `size_t` that specifies the length of the string, the JSON Adapter can
  275. calculate the value of that length parameter automatically because in the
  276. JSON API the lengths of strings are always known.
  277. Moreover, the "length" that has to be given here means the length of the
  278. string seen by the C API side, after processing of all JSON escaping
  279. mechanisms, so it might be difficult to calculate that value at client side.
  280. Example:
  281. ```
  282. // C function declaration:
  283. char* tohex(const char* input, size_t length);
  284. // API definition:
  285. // with implicit length parameter, with dummy JSON parameter
  286. FP( "tohex", new Func<char*, In<c_string>, InLength<>>( &tohex ))
  287. ```
  288. To be compatible with previous API versions the `InLength` parameter still
  289. needs a dummy placeholder in the JSON interface, but its value is no longer
  290. relevant:
  291. ```
  292. {"jsonrpc":"2.0", "id":28,
  293. "method":"tohex", "params":["some string","dummy_parameter"]
  294. }
  295. ```
  296. It is possible to specifiy `InLength<ParamFlag::NoInput>` so no
  297. parameter is exposed to the JSON API anymore:
  298. ```
  299. FP( "tohex", new Func<char*, In<c_string>, InLength<ParamFlag::NoInput>>( &tohex ))
  300. ```
  301. Now the 2nd parameter is omitted:
  302. ```
  303. {"jsonrpc":"2.0", "id":28,
  304. "method":"tohex", "params":["some string"]
  305. }
  306. ```
  307. ## TODOs
  308. The following issues are planned but not yet implemented.
  309. * More sensible unit tests
  310. * Generate all the tedious boiler plate code
  311. * the content of pep-types.cc
  312. * perhaps the FunctionMap 'function' in mt-server.cc
  313. * perhaps the JavaScript side of the HTML test page to ensure to be
  314. consistent with the server side in pep-types.cc
  315. * Adapt the "p≡p Transport API", when it is final. (either manually or by
  316. code generator, if ready)
  317. ## Appendix A: Attack scenarios on the authentication
  318. Let's discuss different attack / threat scenarios. I don't know which are
  319. realistic or possible, yet.
  320. ### General ideas / improvements
  321. Currently the JSON Server Adapter writes its server token file in a
  322. directory that is only readable & writable by the user itself.
  323. The server token file is written in $HOME/.pEp/json-token on
  324. UNIX/Linux/MacOS and %LOCALAPPDATA%/pEp/json-token on MS Windows.
  325. The JSON Server Adapter also checks whether .pEp has 0700 access rights
  326. on unixoid systems.
  327. ### Attacker with the same user rights
  328. If the attacker is able to run his malicious code with the same user
  329. rights as the JSON Server Adapter and his legitimate client, it is (and
  330. always will be) *impossible* to prevent this attack. Such an attacker also
  331. can just start a legitimate client that is under his control.
  332. The same applies to an attacker who gains root / admin access rights.
  333. ### Fake Server with different user rights
  334. ```
  335. ,----------. ,--------.
  336. | Attacker | <==> | Client |
  337. `----------' `--------'
  338. ```
  339. If no real JSON Adapter runs an attacker can create a fake server that
  340. pretends to be a legitimate JSON Adapter. It creates its own server token
  341. file, with different and conspicuous access rights, but a limited
  342. JavaScript client might be unable to detect the file permissions.
  343. This fake server cannot access the private key of the user but it might
  344. get sensitive plaintext data the client wants to encrypt. The fake server
  345. cannot sign the encrypted data so the fake would be conspicuous, too. But
  346. that would be too late, because the sensitive plaintext data could
  347. already be leaked by the fake server.
  348. This attack needs a user's home directory that is writable by someone else
  349. (to create a ~/.pEp/ directory) or a foreign-writable ~/.pEp/ directory.
  350. The pEpEngine creates a ~/.pEp/ directory (if not yet exists) and sets the
  351. permissions to 0700 explicitly.
  352. ### Man-in-the-middle with different user rights
  353. ```
  354. ,---------------------. ,----------. ,--------.
  355. | JSON Server Adapter | <==> | Attacker | <==> | Client |
  356. `---------------------' `----------' `--------'
  357. ```
  358. * The attacker cannot read "client token file" nor "server token file".
  359. * The server cannot check "who" connects to it, until the client
  360. authenticates itself, which might be relayed by the attacker from the
  361. original client.
  362. * The attacker has to convince the client that it is a legitimate server. It
  363. has to create a fake "server token file" to divert the client to the
  364. attacker's port. But that fake file cannot contain the right server token
  365. because the attacker does not know it.
  366. * if the server started before the attacker the "server token file"'s
  367. access rights should prevent this (no write access for the attacker, no
  368. "delete" right in common TEMP dir (sticky bit on the directory)
  369. * if the attacker starts before the server it can write a fake toke file.
  370. The server could detect it but is unable to notice the legitimate
  371. client. The client could detect it when it can check the file access
  372. rights.
  373. There might be race conditons...
  374. * Is it possible for the attacker to let the client send the right server
  375. token to him, at least temporarily (e.g. within a race condition)?
  376. * As long as the server runs, the attacker cannot bind to the same address
  377. & port. Finding and binding of the port is done by the server before the
  378. server token file is created and filled.
  379. * When the server that created the "server token file" dies, its port
  380. becomes available for the attacker, but the server token is no longer
  381. valid and no longer useful for the attacker.
  382. * there _might_ be a very _small_ chance for a race condition:
  383. 1. The attacker opens a connection to the running server but does not
  384. use it. To find the server it cannot read the server configuration
  385. file, but it can ask the OS for ports that are open in "listen" mode.
  386. Normally the JSON Adapter listens on 4223 or some port numbers above
  387. that. That means: guessing the server's address is quite easy.
  388. 2. when the server shuts down, the attacker immediately binds itself to
  389. that port. If a client connects just in this moment it sends the server
  390. token to the attacker, not to the server. But the attacker can use that
  391. token now to the server via the already opened TCP connection.
  392. 3. To prevent this the server should call shutdown(2) on its listening
  393. socket to block any new client connection, but still block the port.
  394. (is that the case on all platforms?) Than close(2) all TCP connections
  395. to the clients (if any) and than also delete the server token file.
  396. Finally call close(2) on the listening socket.