Compare commits

...

42 Commits

Author SHA1 Message Date
  Claudio Luck ea62dd37c2 use system-specific path separators 2 years ago
  Claudio Luck c49c06d8e9 read version from setup.cfg 2 years ago
  Claudio Luck b4780b15b5 Merge 2.6.3 2 years ago
  Claudio Luck cfcde838d9 Merge 2.6.3 2 years ago
  Claudio Luck 548dbedf74 make main(), easier merge 2 years ago
  Claudio Luck 78f926e2c6 ignore setup.py stuff in .hgignore 2 years ago
  Claudio Luck b98d6bb1b7 make PyPI package 2 years ago
  Hartmut Goebel 7e7d1b2934 Release 2.6.3 2 years ago
  Hartmut Goebel a84b0c9f8a Bump version to 2.6.3. 2 years ago
  Hartmut Goebel 661c83c7a8 Update copyright year. 2 years ago
  Claudio Luck 4689784886 fix version 2 years ago
  Claudio Luck a83a87e7be pypackage: Merge 2.6.2 2 years ago
  Hartmut Goebel 371febd7de Makefile: Add target `install`. 6 years ago
  Hartmut Goebel afef6d8dea Makefile: Add target `dist`. 2 years ago
  Hartmut Goebel 9b203022a9 Update .hgignore. 2 years ago
  Claudio Luck e783083273 Add a README. 4 years ago
  Hartmut Goebel 455a941378 Add setup.py, setup.cfg and MANIFEST.in. 2 years ago
  Hartmut Goebel f872862ee9 docs: Include files are always searched at standard location, too. 6 years ago
  Hartmut Goebel 4c7ffe5d0d Move data files into the Python package. 2 years ago
  Hartmut Goebel 4db2ab6fa7 Catch missing xml2yml.ysl2 early. 6 years ago
  Hartmut Goebel af172e42d6 Convert yml2 into a Python package. 2 years ago
  Volker Birk f82481c54e date 3 years ago
  Volker Birk 22f7358623 Added tag 2.6.2 for changeset f93de5b34430 3 years ago
  Volker Birk 01df3fa5e3 explicitely name match attribute 3 years ago
  Volker Birk 2bf92f3d1f documentation 3 years ago
  Volker Birk ffbf8da61d Added tag 2.6.1 for changeset 6c0e2881db60 3 years ago
  Volker Birk 69596b898a bug: textsectionu disappeared 3 years ago
  Volker Birk c5c456145c exit on version string 3 years ago
  Volker Birk b722d7b93c Added tag 2.6.0 for changeset 9b2eff55a3ae 3 years ago
  Volker Birk d3fca48fe5 fixes 3 years ago
  Volker Birk 7f4492af25 adapting to Python 3 3 years ago
  Claudio Luck 4fab8961a9 Track changes of 2.5.10 3 years ago
  Volker Birk 4833362d6b Added tag 2.5.10 for changeset c48f0d1d29ac 3 years ago
  Volker Birk 8650e9a8aa feature: default function is used as a template 3 years ago
  Volker Birk 8213840372 adding hex literals 3 years ago
  Volker Birk d9a2d1888f ... 3 years ago
  Volker Birk b6b4a93110 adding samples 3 years ago
  Claudio Luck df721ef4a7 wip: add setup.py, README.md and debian packaging 4 years ago
  Claudio Luck e905bb7331 rename binaries but add compat symlinks (for prep for setup.py) 4 years ago
  Claudio Luck 59d92156d9 move module body into main() 4 years ago
  Claudio Luck c164ed39c1 prepare to move body code into main function 4 years ago
  Claudio Luck 356f1ca751 move python code into new package dir 4 years ago
36 changed files with 898 additions and 516 deletions
Split View
  1. +5
    -0
      .hgignore
  2. +5
    -0
      .hgtags
  3. +0
    -0
      COPYING.txt
  4. +1
    -0
      LICENSE.txt
  5. +19
    -0
      MANIFEST.in
  6. +41
    -7
      Makefile
  7. +144
    -0
      README.md
  8. +10
    -0
      README.rst
  9. +5
    -0
      debian/changelog
  10. +1
    -0
      debian/compat
  11. +18
    -0
      debian/control
  12. +8
    -0
      debian/rules
  13. +1
    -0
      debian/source/format
  14. +4
    -0
      debian/watch
  15. +1
    -0
      features.en.yhtml2
  16. +3
    -1
      heading.en.yhtml2
  17. +12
    -0
      samples/adresslist.yml2
  18. +6
    -0
      samples/gen.bas
  19. +15
    -0
      samples/gen_addr_sql.ysl2
  20. +0
    -0
      samples/manpage.yml2
  21. +0
    -7
      samples/sample.yml2
  22. +22
    -0
      setup.cfg
  23. +29
    -0
      setup.py
  24. +2
    -2
      toolchain.en.yhtml2
  25. +0
    -0
      yml2/__init__.py
  26. +128
    -124
      yml2/backend.py
  27. +1
    -0
      yml2/manpage.yml2
  28. +30
    -30
      yml2/pyPEG.py
  29. +0
    -0
      yml2/standardlib.ysl2
  30. +0
    -0
      yml2/xml2yml.ysl2
  31. +7
    -7
      yml2/yml2.py
  32. +107
    -0
      yml2/yml2c.py
  33. +253
    -0
      yml2/yml2proc.py
  34. +2
    -2
      yml2/yslt.yml2
  35. +9
    -95
      yml2c
  36. +9
    -241
      yml2proc

+ 5
- 0
.hgignore View File

@ -1,2 +1,7 @@
syntax: glob
*.pyc
*.html
dist/
build/
*.egg-info/
.git/

+ 5
- 0
.hgtags View File

@ -3,3 +3,8 @@ a0f3d678b3cad40e1ea2db938e6b47ad8fa32a3d 2.5.5
b4343a05f5d584cde0656896f4bf2395c805efcb 2.5.7
172da49a35f62a305bfe06287fc6da74822039b0 2.5.8
8138b86a919635f85bd7c8b195d1035ff85e231b 2.5.9
c48f0d1d29ac0dcf8b912fda56702bbbe5431909 2.5.10
9b2eff55a3aeb3fad6b99ff5b1fd17c2cd6e5f6a 2.6.0
6c0e2881db60c4640d36864e2f0007c8fb2df1ae 2.6.1
f93de5b34430e86b9fd9e9012107f91d6f30d116 2.6.2
b307d3f5b0bff0fb706d3a3f7bd1c03369b7c224 2.6.3

gpl-2.0.txt → COPYING.txt View File


+ 1
- 0
LICENSE.txt View File

@ -0,0 +1 @@
See COPTING.txt

+ 19
- 0
MANIFEST.in View File

@ -0,0 +1,19 @@
# Files to include in the source package - for the case
# the project directory is are not in a VCS.
include yml2c
include yml2proc
include yml2/*.ysl2
include yml2/*.yml2
include Makefile
include *.yhtml2
include format.css
include COPYING.txt
include README.rst
include README.md
include samples/*
include vim/syntax/*
exclude .hgtags
exclude .hgignore

+ 41
- 7
Makefile View File

@ -1,20 +1,54 @@
YML_PATH=
YML2C=yml2c
YML_PATH =
DEBVER := 1
PYTHON := python3
PKGVER = $(shell awk '/^version/ {print $$3}' setup.cfg)
TWINE := $(PYTHON) -m twine
TWINEREP := pypi
YML2C = $(PYTHON) yml2c
all: homepage
homepage: index.html features.html yslt.html toolchain.html programming.html hello.html
update: homepage format.css gpl-2.0.txt
rsync -avC *.html *.yml2 format.css *.yhtml2 gpl-2.0.txt samples dragon:fdik.org/yml2/
update: homepage format.css COPYING.txt
rsync -avC *.html *.yml2 format.css *.yhtml2 COPYING.txt samples dragon:fdik.org/yml2/
update-all: update yml2c yml2.py pyPEG.py backend.py yml2proc
if test -z $(VERSION) ; then echo VERSION not set ; exit 1 ; fi
rsync -avC *.py yml2c Makefile yml2proc xml2yml.ysl2 standardlib.ysl2 samples dragon:fdik.org/yml2/
ssh dragon bash -c "cd ; cd fdik.org/; tar cvjf yml-$(VERSION).tar.bz2 yml2/{*.py,*.yml2,*.yhtml2,format.css,gpl-2.0.txt,yml2c,Makefile,yml2proc,xml2yml.ysl2,standardlib.ysl2,samples} ; rm yml2.tar.bz2 ; ln -s yml-$(VERSION).tar.bz2 yml2.tar.bz2"
rsync -avC *.py yml2 yml2c Makefile yml2proc xml2yml.ysl2 standardlib.ysl2 samples dragon:fdik.org/yml2/
ssh dragon bash -c "cd ; cd fdik.org/; tar cvjf yml-$(VERSION).tar.bz2 yml2/{*.py,*.yml2,*.yhtml2,format.css,COPYING.txt,yml2,yml2c,Makefile,yml2proc,xml2yml.ysl2,standardlib.ysl2,samples} ; rm yml2.tar.bz2 ; ln -s yml-$(VERSION).tar.bz2 yml2.tar.bz2"
%.html: %.en.yhtml2 heading.en.yhtml2 homepage.en.yhtml2
$(YML2C) $< -o $@
.PHONY: deb
deb: YML2_$(PKGVER).orig.tar.gz python-yml2_$(PKGVER)-$(DEBVER)_all.deb
YML2_$(PKGVER).orig.tar.gz:
python setup.py sdist
mv -f dist/YML2-$(PKGVER).tar.gz YML2_$(PKGVER).orig.tar.gz
python-yml2_$(PKGVER)-$(DEBVER)_all.deb:
$(PYTHON) setup.py --command-packages=stdeb.command bdist_deb
mv -f deb_dist/python-yml2_$(PKGVER)-$(DEBVER)_all.deb .
clean:
rm -f *.html *.pyc *.pyo
rm -f *.html *.pyc *.pyo */*.pyc */*.pyo
rm -rf *.egg-info
rm -f YML2_$(PKGVER).orig.tar.gz
rm -f python-yml2_$(PKGVER)-$(DEBVER)_all.deb
rm -f dist/YML2-$(PKGVER).tar.gz
sdist-pypi: dist/YML2-$(PKGVER).tar.gz
$(TWINE) check $<
upload-pypi: sdist-pypi
$(TWINE) upload --repository "$(TWINEREP)" $<
.PHONY: dist/YML2-$(PKGVER).tar.gz
dist/YML2-$(PKGVER).tar.gz:
$(PYTHON) setup.py sdist
install:
pip install -e .

+ 144
- 0
README.md View File

@ -0,0 +1,144 @@
## What is YML?
Well, it's the idea not to need to define a grammar first when you want to use a Domain Specific Language. For that purpose, YML is being translated into XML. Let's make an example.
Everything which comes close to a C like language, parses without a grammar definition:
This:
template< class T > T max(T a, T b);
Parses to:
<?xml version='1.0' encoding='UTF-8'?>
<template>
<generic>
<class/>
<T/>
</generic>
<T>
<max>
<parm>
<T/>
<a/>
</parm>
<parm>
<T/>
<b/>
</parm>
</max>
</T>
</template>
Instead of defining grammars, you test out and play around until the results are matching your needs. If the resulting tree does not fit what you're expecting, change it by patching the grammar with `decl`:
This:
module A {
interface B {
attribute long n;
};
};
Parses to:
<?xml version='1.0' encoding='UTF-8'?>
<module>
<A>
<interface>
<B>
<attribute>
<long>
<n/>
</long>
</attribute>
</B>
</interface>
</A>
</module>
This does not look like what we want. So we tell YML that we have a module name after the module, an interface name after the interface and type and name after the attribute:
This:
decl module @name;
decl interface @name;
decl attribute @type @name;
module A {
interface B {
attribute long n;
};
};
Parses to:
<?xml version='1.0' encoding='UTF-8'?>
<module name="A">
<interface name="B">
<attribute type="long" name="n"/>
</interface>
</module>
What can I do with YML?
With YML you can:
* use a C-like DSL without writing a grammar first
* generate code out of this DSL using YSLT
* generate code out of UML using YSLT on XMI
* generate code out of any XML based language like SVG using YSLT
* define a wiki like language in just a few lines like YHTML does
* replace bad designed and complicated XML languages with simpler C-like ones
* ... and much more.
How it works: Replacing angle brackets with some Python
Just writing down what I wanted to have instead of XML for a sample:
<list name="List of goods">
<head>
<columTitle>
Goods
</columnTitle>
<columnTitle>
Price
</columnTitle>
</head>
<row>
<value>
Beer
</value>
<value>
20
</value>
</row>
<row>
<value>
Wine
</value>
<value>
30
</value>
</row>
</list>
Something like that should be more easy, say, like this:
list "List of goods" {
head title "Goods", title "Price";
row value "Beer", value 20;
row value "Wine", value 30;
}
### Y Languages
The latter is what I call an Y language – a language specified in YML. How could this be achieved? Well, what's to do? To have the required information, how to build XML from the script above, we need:
* the information, that “list of goods” is an attribute named `name`, while `Goods` is the text value of a tag
* `title` shout be written out as `columnTitle`
How to do that? Let's invent a simple definition language for that information:
decl list(name);
decl title alias columnTitle;

+ 10
- 0
README.rst View File

@ -0,0 +1,10 @@
************
What is YML?
************
Well, it's the idea not to need to define a grammar first when you want to use a Domain Specific Language. For that purpose, YML is being translated into XML. Let's make an example.
Everything which comes close to a C like language, parses without a grammar definition.
Please see README.md for more information.

+ 5
- 0
debian/changelog View File

@ -0,0 +1,5 @@
yml2 (5.8-1) unstable; urgency=low
* source package automatically created by stdeb 0.8.5
-- Volker Birk <vb@pep.foundation> Tue, 04 Sep 2018 17:29:10 +0200

+ 1
- 0
debian/compat View File

@ -0,0 +1 @@
9

+ 18
- 0
debian/control View File

@ -0,0 +1,18 @@
Source: yml2
Maintainer: Volker Birk <vb@pep.foundation>
Section: python
Priority: optional
Build-Depends: dh-python, python-setuptools (>= 0.6b3), python-all (>= 2.6.6-3), debhelper (>= 9)
Standards-Version: 3.9.6
Homepage: https://pep.foundation/dev/repos/yml2
Package: python-yml2
Architecture: all
Depends: ${misc:Depends}, ${python:Depends}
Description: YML 2 compiler
.
YML2 caters the idea not to need to define a grammar first when you want to
use a Domain Specific Language. For that purpose, YML is being translated into
XML. Let's make an example.

+ 8
- 0
debian/rules View File

@ -0,0 +1,8 @@
#!/usr/bin/make -f
# This file was automatically generated by stdeb 0.8.5 at
# Tue, 04 Sep 2018 17:29:10 +0200
export PYBUILD_NAME=yml2
%:
dh $@ --with python2 --buildsystem=pybuild

+ 1
- 0
debian/source/format View File

@ -0,0 +1 @@
3.0 (quilt)

+ 4
- 0
debian/watch View File

@ -0,0 +1,4 @@
# please also check http://pypi.debian.net/YML2/watch
version=3
opts=uversionmangle=s/(rc|a|b|c)/~$1/ \
http://pypi.debian.net/YML2/YML2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

+ 1
- 0
features.en.yhtml2 View File

@ -975,6 +975,7 @@ page "The Features" {
if the «YML_PATH» environment variable is set to a colon separated list of directories,
these directories are being searched for the given filename. Otherwise, the local
directory is searched.
The system location for «.yml2» and «.ysl2» files is always searched afterwards.
>>
p >>


+ 3
- 1
heading.en.yhtml2 View File

@ -13,10 +13,12 @@ decl ne is a {
div id=headline {
p > YML – Why a Markup Language?!
div class=small {
"YML 2.5.9 of Tu 20 Nov 2018 – Copyleft 2007-2018, "
"YML 2.6.3 of Thu 19 Mar 2020 – Copyleft 2007-2020, "
a "http://blog.fdik.org", "Volker Birk";
" – "
a "http://fdik.org/yml2.tar.bz2", "Download YML 2";
" – "
a "https://pep.foundation/dev/repos/yml2/", "Mercurial repository";
}
}


+ 12
- 0
samples/adresslist.yml2 View File

@ -0,0 +1,12 @@
list {
address {
name "Meier";
street "Uhlmannstrasse 42";
village "Laupheim";
}
address {
name "Schmidt";
street "Haupstrasse 23";
village "Bad Waldsee";
}
}

+ 6
- 0
samples/gen.bas View File

@ -0,0 +1,6 @@
110 DATA "Meier", "Uhlmannstrasse 42", "Laupheim"
120 DATA "Schmidt", "Haupstrasse 23", "Bad Waldsee"
1000 FOR I=1 TO 2 STEP 1
1010 READ NAME$, STREET$, VILLAGE$
1020 PRINT "INSERT INTO address (name, street, village) VALUES ('" + NAME$ + "', '" + STREET$ + "', '" + VILLAGE$ + "');"
1030 NEXT I

+ 15
- 0
samples/gen_addr_sql.ysl2 View File

@ -0,0 +1,15 @@
include yslt.yml2
tstylesheet {
template "/list" {
apply "address", 0;
| 1000 FOR I=1 TO 2 STEP 1
| 1010 READ NAME$, STREET$, VILLAGE$
| 1020 PRINT "INSERT INTO address (name, street, village) VALUES ('" + NAME$ + "', '" + STREET$ + "', '" + VILLAGE$ + "');"
| 1030 NEXT I
}
template "address" {
| «position() * 10 + 100» DATA "«name»", "«street»", "«village»"
}
}

manpage.yml2 → samples/manpage.yml2 View File


+ 0
- 7
samples/sample.yml2 View File

@ -1,7 +0,0 @@
include idl.yml2
module blub {
interface bla {
int f(in long a, out long b);
}
}

+ 22
- 0
setup.cfg View File

@ -0,0 +1,22 @@
[sdist]
formats=gztar
[metadata]
name = yml2
version = 2.6.4
author = Volker Birk
author_email = vb@pep-project.org
url = https://pep.foundation/dev/repos/yml2/
license = GPL-2.0
license_file = COPYING.txt
description-file = README.rst
[options]
packages = yml2
scripts =
yml2c
yml2proc
include_package_data = True
zip_safe = False
install_requires = lxml

+ 29
- 0
setup.py View File

@ -0,0 +1,29 @@
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages
with open('README.rst', 'r', encoding='utf-8') as f:
long_description = f.read()
setup(
setup_requires = ["setuptools >= 30.3.0"], # release 2016-12-06
long_description=long_description,
entry_points = {
'console_scripts': [
'yml2c=yml2.yml2c:main',
'yml2proc=yml2.yml2proc:main'
],
},
classifiers=[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Environment :: Console',
'Topic :: Software Development :: Build Tools',
'License :: OSI Approved :: GNU General Public License v2 (GPLv2)',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
],
)

+ 2
- 2
toolchain.en.yhtml2 View File

@ -87,6 +87,7 @@ page "The YML Toolchain" {
If «YML_PATH» is set to a colon separated ¬http://en.wikipedia.org/wiki/Path_(computing) path¬
list, then the `a href="features#including" code "include";` statement searches these paths.
If «YML_PATH» is not set, the local directory «.» is searched for files.
The system location for «.yml2» and «.ysl2» files is always searched afterwards.
>>
h2 id=processor > The YML/YSLT processor
@ -218,8 +219,6 @@ page "The YML Toolchain" {
p > Convert XML to normalized YML. Equivalent to -xy xml2yml.ysl2
p > The file xml2yml.ysl2 has to be in YML_PATH.
h3 > -V, --version
p >>
@ -233,6 +232,7 @@ page "The YML Toolchain" {
If «YML_PATH» is set to a colon separated ¬http://en.wikipedia.org/wiki/Path_(computing) path¬
list, then the `a href="features#including" code "include";` statement searches these paths.
If «YML_PATH» is not set, the local directory «.» is searched for files.
The system location for «.yml2» and «.ysl2» files is always searched afterwards.
>>
h2 id=recommended > Recommended external tools


+ 0
- 0
yml2/__init__.py View File


backend.py → yml2/backend.py View File


+ 1
- 0
yml2/manpage.yml2 View File

@ -0,0 +1 @@
decl manpage(xmlns="http://docbook.org/ns/docbook", version="5.0", *title) alias article { title *title; content; };

pyPEG.py → yml2/pyPEG.py View File


standardlib.ysl2 → yml2/standardlib.ysl2 View File


xml2yml.ysl2 → yml2/xml2yml.ysl2 View File


yml2.py → yml2/yml2.py View File


+ 107
- 0
yml2/yml2c.py View File

@ -0,0 +1,107 @@
#!/usr/bin/env python3
# vim: set fileencoding=utf-8 :
"""\
YML 2 compiler version 6.2
Copyleft (c), 2009-2020, Volker Birk http://fdik.org/yml/
"""
import sys, os, codecs, locale
import fileinput, unicodedata
from optparse import OptionParser
if __name__ == "__main__":
sys.path.insert(0, os.path.dirname(__file__))
from yml2.pyPEG import parse, u
from yml2.yml2 import ymlCStyle, comment, oldSyntax
import yml2.backend as backend
YML_DEFAULT_PATHS = [os.path.dirname(backend.__file__)]
def printInfo(option, opt_str, value, parser):
sys.stdout.write(__doc__)
sys.exit(0)
def w(msg):
if isinstance(msg, BaseException):
msg = str(msg) + "\n"
if type(msg) is bytes:
msg = codecs.encode(msg, sys.stderr.encoding)
sys.stderr.write(msg)
def main():
dirsepname = {':': 'a colon', ';': 'a semicolon'}.get(os.path.pathsep, "a '%s'" % os.path.pathsep)
optParser = OptionParser()
optParser.add_option("-C", "--old-syntax", action="store_true", dest="old_syntax",
help="syntax of YML 2 version 1.x (compatibility mode)", default=False)
optParser.add_option("-D", "--emit-linenumbers", action="store_true", dest="emitlinenumbers",
help="emit line numbers into the resulting XML for debugging purposes", default=False)
optParser.add_option("-E", "--encoding", dest="encoding", metavar="ENCODING", default=locale.getdefaultlocale()[1],
help="encoding of input files (default to locale)")
optParser.add_option("-I", "--include", dest="includePathText", metavar="INCLUDE_PATH",
help="precede YML_PATH by %s separated INCLUDE_PATH to search for include files" % dirsepname)
optParser.add_option("-m", "--omit-empty-parm-tags", action="store_true", dest="omitemptyparm",
help="does nothing (only there for compatibility reasons)", default=False)
optParser.add_option("-n", "--normalization", dest="normalization", metavar="NORMALIZATION", default="NFC",
help="Unicode normalization (none, NFD, NFKD, NFC, NFKC, FCD, default is NFC)")
optParser.add_option("-o", "--output", dest="outputFile", metavar="FILE",
help="place output in file FILE")
optParser.add_option("-p", "--parse-only", action="store_true", dest="parseonly",
help="parse only, then output pyAST as text to stdout", default=False)
optParser.add_option("-V", "--version", action="callback", callback=printInfo, help="show version info")
(options, args) = optParser.parse_args()
if options.old_syntax:
oldSyntax()
if options.emitlinenumbers:
backend.emitlinenumbers = True
backend.encoding = options.encoding
try:
if options.includePathText:
backend.includePath = options.includePathText.split(os.path.pathsep)
dirs = os.environ.get('YML_PATH', '.').split(os.path.pathsep) + YML_DEFAULT_PATHS
backend.includePath.extend(dirs)
files = fileinput.input(args, mode="rU", openhook=fileinput.hook_encoded(options.encoding))
ymlC = ymlCStyle()
result = parse(ymlC, files, True, comment, packrat=True)
if options.parseonly:
print(result)
else:
result = backend.finish(result)
if options.normalization != "none":
result = unicodedata.normalize(options.normalization, result)
if options.outputFile and options.outputFile != "-":
outfile = open(options.outputFile, "wb")
outfile.write(codecs.encode(result, options.encoding))
outfile.close()
else:
sys.stdout.buffer.write(codecs.encode(result, options.encoding))
print()
except KeyboardInterrupt:
w("\n")
sys.exit(1)
except KeyError as msg:
w("not found: " + u(msg) + "\n")
sys.exit(4)
except LookupError as msg:
w("not found: " + u(msg) + "\n")
sys.exit(4)
except Exception as msg:
w(msg)
sys.exit(5)
if __name__ == "__main__":
sys.exit(main())

+ 253
- 0
yml2/yml2proc.py View File

@ -0,0 +1,253 @@
#!/usr/bin/env python3
# vim: set fileencoding=utf-8 :
"""\
YML/YSLT 2 processor version 6.2
Copyleft (c), 2009-2020 Volker Birk http://fdik.org/yml/
"""
import sys, os, codecs, locale
import fileinput, unicodedata
from optparse import OptionParser
try:
from lxml import etree
except:
sys.stderr.write("This program needs lxml, see http://codespeak.net/lxml/\n")
sys.exit(1)
if __name__ == "__main__":
sys.path.insert(0, os.path.dirname(__file__))
from yml2.yml2 import ymlCStyle, comment, oldSyntax
from yml2.pyPEG import parse, u
import yml2.backend as backend
YML_DEFAULT_PATHS = [os.path.dirname(backend.__file__)]
def printInfo(option, opt_str, value, parser):
sys.stdout.write(__doc__)
sys.exit(0)
class YMLAssert(Exception): pass
def w(msg):
if isinstance(msg, BaseException):
try:
msg = str(msg) + "\n"
except:
msg = u(msg) + "\n"
sys.stderr.write(msg)
def main():
dirsepname = {':': 'a colon', ';': 'a semicolon'}.get(os.path.pathsep, "a '%s'" % os.path.pathsep)
optParser = OptionParser()
optParser.add_option("-C", "--old-syntax", action="store_true", dest="old_syntax",
help="syntax of YML 2 version 1.x (compatibility mode)", default=False)
optParser.add_option("-D", "--emit-linenumbers", action="store_true", dest="emitlinenumbers",
help="emit line numbers into the resulting XML for debugging purposes", default=False)
optParser.add_option("--debug", action="store_true", dest="trace",
help="switch on tracing to stderr", default=False)
optParser.add_option("-d", "--paramdict", dest="params", metavar="PARAMS",
help="call X/YSLT script with dictionary PARAMS as parameters")
optParser.add_option("-e", "--xpath", dest="xpath", metavar="XPATH",
help="execute XPath expression XPATH and print result")
optParser.add_option("-E", "--encoding", dest="encoding", metavar="ENCODING", default=locale.getdefaultlocale()[1],
help="encoding of input files (default to locale)")
optParser.add_option("-I", "--include", dest="includePathText", metavar="INCLUDE_PATH",
help="precede YML_PATH by %s separated INCLUDE_PATH to search for include files" % dirsepname)
optParser.add_option("-m", "--omit-empty-parm-tags", action="store_true", dest="omitemptyparm",
help="does nothing (only there for compatibility reasons)", default=False)
optParser.add_option("-M", "--empty-input-document", action="store_true", dest="emptyinput",
help="use an empty input document", default=False)
optParser.add_option("-n", "--normalization", dest="normalization", metavar="NORMALIZATION", default="NFC",
help="Unicode normalization (none, NFD, NFKD, NFC, NFKC, FCD, default is NFC)")
optParser.add_option("-o", "--output", dest="outputFile", metavar="FILE",
help="place output in file FILE")
optParser.add_option("-p", "--parse-only", action="store_true", dest="parseonly",
help="parse only, then output pyAST as text to stdout", default=False)
optParser.add_option("-P", "--pretty", action="store_true", default=False,
help="pretty print output adding whitespace")
optParser.add_option("-s", "--stringparamdict", dest="stringparams", metavar="STRINGPARAMS",
help="call X/YSLT script with dictionary STRINGPARAMS as string parameters")
optParser.add_option("-x", "--xml", action="store_true", default=False,
help="input document is XML already")
optParser.add_option("-X", "--xslt", dest="xslt", metavar="XSLTSCRIPT",
help="execute XSLT script XSLTSCRIPT")
optParser.add_option("-y", "--yslt", dest="yslt", metavar="YSLTSCRIPT",
help="execute YSLT script YSLTSCRIPT")
optParser.add_option("-Y", "--xml2yml", action="store_true", default=False,
help="convert XML to normalized YML code")
optParser.add_option("-V", "--version", action="callback", callback=printInfo, help="show version info and exit")
(options, args) = optParser.parse_args()
if options.old_syntax:
oldSyntax()
if options.trace:
backend.enable_tracing = True
if options.emitlinenumbers:
backend.emitlinenumbers = True
if options.includePathText:
backend.includePath = options.includePathText.split(os.path.pathsep)
backend.encoding = options.encoding
dirs = os.environ.get('YML_PATH', '.').split(os.path.pathsep) + YML_DEFAULT_PATHS
backend.includePath.extend(dirs)
if options.xml2yml:
for directory in backend.includePath:
name = os.path.join(directory, "xml2yml.ysl2")
if os.path.isfile(name):
options.yslt = name
options.xml = True
break
else:
sys.stderr.write("Error: Stylesheet xml2yml.ysl2 required for --xml2yml not found\n")
sys.stderr.write("Please check your YML_PATH\n")
sys.exit(1)
if (options.xslt and options.yslt) or (options.xslt and options.xpath) or (options.yslt and options.xpath):
sys.stderr.write("Cannot combine --xpath, --xslt and --yslt params\n")
sys.exit(1)
try:
ymlC = ymlCStyle()
rtext = ""
if not options.emptyinput:
files = fileinput.input(args, mode="rU", openhook=fileinput.hook_encoded(options.encoding))
if options.xml:
rtext = ""
for line in files:
rtext += line
else:
result = parse(ymlC, files, True, comment)
if options.parseonly:
print(result)
sys.exit(0)
else:
rtext = backend.finish(result)
if not rtext:
rtext = "<empty/>"
def ymldebug(context, text):
if options.trace:
sys.stderr.write("Debug: " + codecs.encode(u(text), options.encoding) + "\n")
return ""
def ymlassert(context, value, msg):
if options.trace:
if not value:
raise YMLAssert(msg)
return ""
ymlns = etree.FunctionNamespace("http://fdik.org/yml")
ymlns.prefix = "yml"
ymlns['debug'] = ymldebug
ymlns['assert'] = ymlassert
if options.xpath:
tree = etree.fromstring(rtext)
ltree = tree.xpath(codecs.decode(options.xpath, options.encoding))
rtext = ""
try:
for rtree in ltree:
rtext += etree.tostring(rtree, pretty_print=options.pretty, encoding=unicode)
except:
rtext = ltree
elif options.yslt or options.xslt:
params = {}
if options.yslt:
backend.clearAll()
yscript = fileinput.input(options.yslt, mode="rU", openhook=fileinput.hook_encoded(options.encoding))
yresult = parse(ymlC, yscript, True, comment)
ytext = backend.finish(yresult)
else:
yscript = fileinput.input(options.xslt, mode="rU")
ytext = ""
for line in yscript:
ytext += line
doc = etree.fromstring(rtext)
xsltree = etree.XML(ytext, base_url=os.path.abspath(yscript.filename()))
transform = etree.XSLT(xsltree)
if options.params:
params = eval(options.params)
for key, value in params.iteritems():
if type(value) is not str:
params[key] = u(value)
if options.stringparams:
for key, value in eval(options.stringparams).iteritems():
params[key] = "'" + u(value) + "'"
rresult = transform(doc, **params)
# lxml is somewhat buggy
try:
rtext = u(rresult)
except:
rtext = etree.tostring(rresult, encoding=unicode)
if not rtext:
rtext = codecs.decode(str(rresult), "utf-8")
if options.normalization != "none":
rtext = unicodedata.normalize(options.normalization, rtext)
if options.pretty:
plaintext = etree.tostring(etree.fromstring(rtext), pretty_print=True, xml_declaration=True, encoding=options.encoding)
else:
if isinstance(rtext, str):
plaintext = codecs.encode(rtext, options.encoding)
else:
plaintext = rtext
try:
if plaintext[-1] == "\n":
plaintext = plaintext[:-1]
except: pass
if options.outputFile and options.outputFile != "-":
outfile = open(options.outputFile, "wb")
outfile.write(plaintext)
outfile.close()
else:
sys.stdout.buffer.write(plaintext)
if not options.pretty:
print()
except KeyboardInterrupt:
w("\n")
sys.exit(1)
except YMLAssert as msg:
w("YML Assertion failed: " + u(msg) + "\n")
sys.exit(2)
except KeyError as msg:
w("not found: " + u(msg) + "\n")
sys.exit(4)
except LookupError as msg:
w("not found: " + u(msg) + "\n")
sys.exit(4)
except etree.XMLSyntaxError as e:
log = e.error_log.filter_from_level(etree.ErrorLevels.FATAL)
for entry in log:
w("XML error: " + u(entry.message) + "\n")
sys.exit(5)
except Exception as msg:
w(msg)
sys.exit(5)
if __name__ == "__main__":
sys.exit(main())

yslt.yml2 → yml2/yslt.yml2 View File


+ 9
- 95
yml2c View File

@ -1,96 +1,10 @@
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
"""\
YML 2 compiler version 5.9
Copyleft (c), 2009-2018, Volker Birk http://fdik.org/yml/
"""
import sys, os, codecs, locale
import fileinput, unicodedata
from optparse import OptionParser
from pyPEG import parse, u
from yml2 import ymlCStyle, comment, oldSyntax
import backend
def printInfo(option, opt_str, value, parser):
sys.stdout.write(__doc__)
def w(msg):
if isinstance(msg, BaseException):
try:
msg = str(msg) + "\n"
except:
msg = u(msg) + u"\n"
if type(msg) is unicode:
msg = codecs.encode(msg, sys.stderr.encoding)
sys.stderr.write(msg)
optParser = OptionParser()
optParser.add_option("-C", "--old-syntax", action="store_true", dest="old_syntax",
help="syntax of YML 2 version 1.x (compatibility mode)", default=False)
optParser.add_option("-D", "--emit-linenumbers", action="store_true", dest="emitlinenumbers",
help="emit line numbers into the resulting XML for debugging purposes", default=False)
optParser.add_option("-E", "--encoding", dest="encoding", metavar="ENCODING", default=locale.getdefaultlocale()[1],
help="encoding of input files (default to locale)")
optParser.add_option("-I", "--include", dest="includePathText", metavar="INCLUDE_PATH",
help="precede YML_PATH by a colon separated INCLUDE_PATH to search for include files")
optParser.add_option("-m", "--omit-empty-parm-tags", action="store_true", dest="omitemptyparm",
help="does nothing (only there for compatibility reasons)", default=False)
optParser.add_option("-n", "--normalization", dest="normalization", metavar="NORMALIZATION", default="NFC",
help="Unicode normalization (none, NFD, NFKD, NFC, NFKC, FCD, default is NFC)")
optParser.add_option("-o", "--output", dest="outputFile", metavar="FILE",
help="place output in file FILE")
optParser.add_option("-p", "--parse-only", action="store_true", dest="parseonly",
help="parse only, then output pyAST as text to stdout", default=False)
optParser.add_option("-V", "--version", action="callback", callback=printInfo, help="show version info")
(options, args) = optParser.parse_args()
if options.old_syntax:
oldSyntax()
if options.emitlinenumbers:
backend.emitlinenumbers = True
backend.encoding = options.encoding
#!/usr/bin/env python3
import sys, os
sys.path.insert(0, os.path.dirname(__file__))
try:
if options.includePathText:
backend.includePath = options.includePathText.split(':')
dirs = os.environ.get('YML_PATH', '.').split(':')
backend.includePath.extend(dirs)
files = fileinput.input(args, mode="rU", openhook=fileinput.hook_encoded(options.encoding))
ymlC = ymlCStyle()
result = parse(ymlC, files, True, comment, packrat=True)
if options.parseonly:
print(result)
else:
result = backend.finish(result)
if options.normalization != "none":
result = unicodedata.normalize(options.normalization, result)
if options.outputFile and options.outputFile != "-":
outfile = open(options.outputFile, "w")
outfile.write(codecs.encode(result, options.encoding))
outfile.close()
else:
print(codecs.encode(result, options.encoding))
except KeyboardInterrupt:
w("\n")
sys.exit(1)
except KeyError as msg:
w(u"not found: " + u(msg) + u"\n")
sys.exit(4)
except LookupError as msg:
w(u"not found: " + u(msg) + u"\n")
sys.exit(4)
except Exception as msg:
w(msg)
sys.exit(5)
from yml2 import yml2c
except ImportError as e:
import pkg_resources
pkg_resources.require('YML2')
from yml2 import yml2c
sys.exit(yml2c.main())

+ 9
- 241
yml2proc View File

@ -1,242 +1,10 @@
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
"""\
YML/YSLT 2 processor version 5.9
Copyleft (c), 2009-2018 Volker Birk http://fdik.org/yml/
"""
import sys, os, codecs, locale
import fileinput, unicodedata
from optparse import OptionParser
#!/usr/bin/env python3
import sys, os
sys.path.insert(0, os.path.dirname(__file__))
try:
from lxml import etree
except:
sys.stderr.write("This program needs lxml, see http://codespeak.net/lxml/\n")
sys.exit(1)
from yml2 import ymlCStyle, comment, oldSyntax
from pyPEG import parse, u
import backend
def printInfo(option, opt_str, value, parser):
sys.stdout.write(__doc__)
sys.exit(0)
class YMLAssert(Exception): pass
def w(msg):
if isinstance(msg, BaseException):
try:
msg = str(msg) + "\n"
except:
msg = u(msg) + u"\n"
if type(msg) is unicode:
msg = codecs.encode(msg, sys.stderr.encoding)
sys.stderr.write(msg)
optParser = OptionParser()
optParser.add_option("-C", "--old-syntax", action="store_true", dest="old_syntax",
help="syntax of YML 2 version 1.x (compatibility mode)", default=False)
optParser.add_option("-D", "--emit-linenumbers", action="store_true", dest="emitlinenumbers",
help="emit line numbers into the resulting XML for debugging purposes", default=False)
optParser.add_option("--debug", action="store_true", dest="trace",
help="switch on tracing to stderr", default=False)
optParser.add_option("-d", "--paramdict", dest="params", metavar="PARAMS",
help="call X/YSLT script with dictionary PARAMS as parameters")
optParser.add_option("-e", "--xpath", dest="xpath", metavar="XPATH",
help="execute XPath expression XPATH and print result")
optParser.add_option("-E", "--encoding", dest="encoding", metavar="ENCODING", default=locale.getdefaultlocale()[1],
help="encoding of input files (default to locale)")
optParser.add_option("-I", "--include", dest="includePathText", metavar="INCLUDE_PATH",
help="precede YML_PATH by a colon separated INCLUDE_PATH to search for include files")
optParser.add_option("-m", "--omit-empty-parm-tags", action="store_true", dest="omitemptyparm",
help="does nothing (only there for compatibility reasons)", default=False)
optParser.add_option("-M", "--empty-input-document", action="store_true", dest="emptyinput",
help="use an empty input document", default=False)
optParser.add_option("-n", "--normalization", dest="normalization", metavar="NORMALIZATION", default="NFC",
help="Unicode normalization (none, NFD, NFKD, NFC, NFKC, FCD, default is NFC)")
optParser.add_option("-o", "--output", dest="outputFile", metavar="FILE",
help="place output in file FILE")
optParser.add_option("-p", "--parse-only", action="store_true", dest="parseonly",
help="parse only, then output pyAST as text to stdout", default=False)
optParser.add_option("-P", "--pretty", action="store_true", default=False,
help="pretty print output adding whitespace")
optParser.add_option("-s", "--stringparamdict", dest="stringparams", metavar="STRINGPARAMS",
help="call X/YSLT script with dictionary STRINGPARAMS as string parameters")
optParser.add_option("-x", "--xml", action="store_true", default=False,
help="input document is XML already")
optParser.add_option("-X", "--xslt", dest="xslt", metavar="XSLTSCRIPT",
help="execute XSLT script XSLTSCRIPT")
optParser.add_option("-y", "--yslt", dest="yslt", metavar="YSLTSCRIPT",
help="execute YSLT script YSLTSCRIPT")
optParser.add_option("-Y", "--xml2yml", action="store_true", default=False,
help="convert XML to normalized YML code")
optParser.add_option("-V", "--version", action="callback", callback=printInfo, help="show version info and exit")
(options, args) = optParser.parse_args()
if options.old_syntax:
oldSyntax()
if options.trace:
backend.enable_tracing = True
if options.emitlinenumbers:
backend.emitlinenumbers = True
if options.includePathText:
backend.includePath = options.includePathText.split(':')
backend.encoding = options.encoding
dirs = os.environ.get('YML_PATH', '.').split(':')
backend.includePath.extend(dirs)
if options.xml2yml:
for directory in backend.includePath:
try:
name = directory + "/xml2yml.ysl2"
f = open(name, "r")
f.close()
break
except:
pass
options.yslt = name
options.xml = True
if (options.xslt and options.yslt) or (options.xslt and options.xpath) or (options.yslt and options.xpath):
sys.stderr.write("Cannot combine --xpath, --xslt and --yslt params\n")
sys.exit(1)
try:
ymlC = ymlCStyle()
rtext = u""
if not options.emptyinput:
files = fileinput.input(args, mode="rU", openhook=fileinput.hook_encoded(options.encoding))
if options.xml:
rtext = ""
for line in files:
rtext += line
else:
result = parse(ymlC, files, True, comment)
if options.parseonly:
print(result)
sys.exit(0)
else:
rtext = backend.finish(result)
if not rtext:
rtext = u"<empty/>"
def ymldebug(context, text):
if options.trace:
sys.stderr.write("Debug: " + codecs.encode(u(text), options.encoding) + "\n")
return ""
def ymlassert(context, value, msg):
if options.trace:
if not value:
raise YMLAssert(msg)
return ""
ymlns = etree.FunctionNamespace("http://fdik.org/yml")
ymlns.prefix = "yml"
ymlns['debug'] = ymldebug
ymlns['assert'] = ymlassert
if options.xpath:
tree = etree.fromstring(rtext)
ltree = tree.xpath(codecs.decode(options.xpath, options.encoding))
rtext = u""
try:
for rtree in ltree:
rtext += etree.tostring(rtree, pretty_print=options.pretty, encoding=unicode)
except:
rtext = ltree
elif options.yslt or options.xslt:
params = {}
if options.yslt:
backend.clearAll()
yscript = fileinput.input(options.yslt, mode="rU", openhook=fileinput.hook_encoded(options.encoding))
yresult = parse(ymlC, yscript, True, comment)
ytext = backend.finish(yresult)
else:
yscript = fileinput.input(options.xslt, mode="rU")
ytext = ""
for line in yscript:
ytext += line
doc = etree.fromstring(rtext)
xsltree = etree.XML(ytext, base_url=os.path.abspath(yscript.filename()))
transform = etree.XSLT(xsltree)
if options.params:
params = eval(options.params)
for key, value in params.iteritems():
if type(value) != unicode:
params[key] = u(value)
if options.stringparams:
for key, value in eval(options.stringparams).iteritems():
params[key] = u"'" + u(value) + u"'"
rresult = transform(doc, **params)
# lxml is somewhat buggy
try:
rtext = u(rresult)
except:
rtext = etree.tostring(rresult, encoding=unicode)
if not rtext:
rtext = codecs.decode(str(rresult), "utf-8")
if options.normalization != "none":
rtext = unicodedata.normalize(options.normalization, rtext)
if options.pretty:
plaintext = etree.tostring(etree.fromstring(rtext), pretty_print=True, xml_declaration=True, encoding=options.encoding)
else:
if isinstance(rtext, unicode):
plaintext = codecs.encode(rtext, options.encoding)
else:
plaintext = str(rtext)
try:
if plaintext[-1] == "\n":
plaintext = plaintext[:-1]
except: pass
if options.outputFile and options.outputFile != "-":
outfile = open(options.outputFile, "w")
outfile.write(plaintext)
outfile.close()
else:
print(plaintext)
except KeyboardInterrupt:
w("\n")
sys.exit(1)
except YMLAssert as msg:
w(u"YML Assertion failed: " + u(msg) + u"\n")
sys.exit(2)
except KeyError as msg:
w(u"not found: " + u(msg) + u"\n")
sys.exit(4)
except LookupError as msg:
w(u"not found: " + u(msg) + u"\n")
sys.exit(4)
except etree.XMLSyntaxError as e:
log = e.error_log.filter_from_level(etree.ErrorLevels.FATAL)
for entry in log:
w(u"XML error: " + u(entry.message) + u"\n")
sys.exit(5)
except Exception as msg:
w(msg)
sys.exit(5)
from yml2 import yml2proc
except ModuleNotFoundError:
import pkg_resources
pkg_resources.require('YML2')
from yml2 import yml2proc
sys.exit(yml2proc.main())

Loading…
Cancel
Save