docutils-develop Mailing List for Docutils: Documentation Utilities (Page 415)

On Mon, 23 Sep 2002 22:50:32 -0400 David Goodger
<go...@us...> wrote:

[...]
> Opinions?  Is this necessary or useful?  Should we move existing
> projects or just new ones?  Would anyone like to volunteer for their
> project to move over?  Does anyone object to this approach?

I think it seems like a good idea, and moving old projects seems natural when
more people start working on a single one...

---
Adam Chodorowski <ad...@ch...>

The way I understand it, the Russians are sort of a combination of evil and
incompetence... sort of like the Post Office with tanks.
    -- Emo Philips

Hi,

David Goodger:
> I believe the difficulty in TeX is that a footnote is assumed to have only
> one reference.  I.e., you can't say [1]_ and then say [1]_ again somewhere
> else.  I'll leave it to those with real TeX knowledge (i.e., not me!) to
> figure out a solution.

This will happen very seldom, so I propose the following solution:

Emit a LaTeX 

\footnote{\label{<FOOTNOTE_IDENTIFIER>}FOOTNOTE_TEXT} 

macro at the first encountered reference.  At the second, third and so on
reference emit a LaTeX

\footnote{see footnote \ref{FOOTNOTE_IDENTIFIER} on 
page \pagref{FOOTNOTE_IDENTIFIER}.}

This is not perfect but would deal with the situation in a reasonable way.

The LaTeX writer should also pay attention to the fact, that
standard LaTeX does not allow to construct footnotes inside tabular
material.  I advise to use the tabularx environment and to add a
\usepackage{tabularx} into the header section.  This shouldn't be a
problem, since tabularx may be considered as Standard LaTeX today:
It was even documented in LaTeX companion from 1994.

Using the LaTeX writer and pdflatex would offer a decent
reST to PDF converter.

Regards, Peter
-- 
Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany
office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen, Germany)

There sandbox is becoming an interesting place.  There are several
useful applications of Docutils (ZReST, PythonPoint writer,
DocFactory), the LaTeX writer has been picked up by a second
developer, and the PySource writer is up for grabs.  It seems to me
that these projects have outgrown the developer-oriented sandbox.

With the possibility of two or more people working on the same sandbox
project, perhaps it's time to re-think the sandbox organization.
Sandbox projects are inside directories with peoples' name on them.
This may give people pause, thinking that the projects are "owned" and
off-limits, when in fact they should be open and inviting.  A
half-finished project is a sad thing, and we should encourage
participation from anyone who's interested.

I think projects ought to be emphasized, rather than people.  Perhaps
we should create a new top-level "projects" directory, open to
participation.  A README.txt file at the top of each project could
identify the project coordinator and outline any requirements (like
"please run patches by me before checking them in").  Some of the
material in the sandbox is not project-oriented though, and should
remain in the sandbox.

When we move a directory, the continuity of the CVS history of the
files will be broken unless we ask the SourceForge staff to move the
directories for us.  They appear to be willing to do this (they've
done individual files for me in the past), but I'd rather ask all at
once than in dribs and drabs, so we'll need an organized plan.

Opinions?  Is this necessary or useful?  Should we move existing
projects or just new ones?  Would anyone like to volunteer for their
project to move over?  Does anyone object to this approach?

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Engelbert, your mail to your ssg.co.at account is bouncing (forwarded from
users.sf.net).

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Richard Jones wrote:
> Can't you search the DOM for the footnote body, based on the reference?

In fact, all footnotes are kept track of while processing the document.
``document.footnotes``, ``document.footnote_refs``,
``document.autofootnotes``, etc.  Take a look at docutils/nodes.py, ``class
document``, for details.  Use the "--dump-internals" command-line option to
see what's stored where.

> If not, can you maybe have a simple transform that runs pre-writer that
> replaces footnote ref/footnote with a single footnote node?

I believe the difficulty in TeX is that a footnote is assumed to have only
one reference.  I.e., you can't say [1]_ and then say [1]_ again somewhere
else.  I'll leave it to those with real TeX knowledge (i.e., not me!) to
figure out a solution.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Adam Chodorowski wrote:
> In the project I'm using docutils in, we've decided to convert the
> ReST sources to XML (with the docutils DTD) and then use some XSLT
> tools to transform this into different formats. [#]_
...
> .. [#] The reason for this, as opposed to writing a new docutils
>        writer, is mainly because we have a developer that has a lot
>        of experience with XML/XSLT and no experience with docutils
>        or Python: we believe that this aproach will give us results
>        faster, basically.

That's fair and reasonable.  Whatever works!

> However, we discovered that the output from "quicktest.py -r" is not
> valid XML.

That's probably not the best tool to use.  Note the name,
"quicktest.py".  It's meant for quick & dirty output for parser
testing purposes.  There's no Reader or Writer involved, and no
transforms are applied.  See
http://docutils.sf.net/docs/tools.html#quicktest-py .

> What's the best way to fix these issues? Perhaps a dedicated XML
> writer is necessary (I hope not...)?

One already exists: docutils-xml.py.  Try it instead of quicktest.py.

I just tried running it on tools/test.txt and had some trouble.  Fell
victim to a bug in Python 2.2's StringIO.py.  Moral: upgrade to 2.2.1!
With 2.2.1 in place, it worked without problem.

I've added an encoding attribute to the XML declaration (``<?xml
version="1.0" encoding="utf-8"?>``) and a ``<!DOCTYPE document ...``
declaration.  It would be easy enough to add an XSL stylesheet
declaration as well; I'll wait for the need to arise.

> There are two problems with the output:
> 
> 1. The text contents are not encoded / decoded in any way;
>    everything is just output verbatim. This is a problem if the
>    input file is ISO-8859-1 and XML is supposed to be UTF-8. Some
>    --input-encoding and --output-encoding options would be nice.

Patches are welcome, but I suspect that once you try docutils-xml.py
you'll forget all about quicktest.py.

> 2. The "xml:space" attribute is used incorrectly. In the output one
>    can find "xml:space=1"; the only valid values from xml:space are
>    "default" and "preserved".

An oversight, now fixed ("preserve", actually).  Thanks.  The snapshot
has been updated: http://docutils.sf.net/docutils-snapshot.tgz

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

[David]
>>> Engelbert, are you planning on continuing development of the LaTeX
>>> writer or the PDF writer in the near future?  Or would it be OK if
>>> Julien or someone else took over?

[Engelbert]
> i am willing to continue, but i have no time for theoretical work
> means the writer is finished as sson as it fullfills what is needed

Let me presume to state that what is needed is to completely and
correctly process the tools/test.txt file.  That's all! ;-)

> and i can see some problem s coming up, interesting for all the
> project, because this would be the second writer, but bigger ones,
> because the first writer and the whole thing is targeted at html.

Docutils is not targeted at HTML.  It's just that the HTML writer is
the only one of real significance.  The XML writer is trivial, and
easily ignored.  Until there's another major Writer, it's easy to
understand the mistaken perception that Docutils is HTML-oriented.

> html for instances has no page length and no pagewidth, latex needs
> this.

The LaTeX writer can define its own command-line options and config
file settings.

> actually i currently have a project where docutils with a latex
> writer might help me, so i will continue.

Great!  I hope you and Julien can work together on this.  It would be
wonderful to get another major Writer in Docutils.

> rlpdf is on hold for several reasons
> * no one needs it currently

I don't know about that.  If it existed, I'm sure there would be
users.  I've seen requests several times for PDF output.  However:

> * pdflatex should do it and would do it better as developing the
>   linebreaking in python/ps seams to be heavy.

If that's a better route, then by all means pursue it.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

On Tue, 24 Sep 2002 12:38 am, Julien T. Letessier wrote:
> Hey list, it's me again.
>
> On Sep 23, 2002, at 15:27, gr...@us...  was heard saying:
> > you could send me your version for review ?
>
> No problem.
>
> > > * the problem comes from the fact that footnote refs and footnote
> > > bodies are separated in ReST, whereas in LaTeX, they're not.
> > >
> > > * my first solution doesn't work (defining footnote bodies in the
> > > document head, and referring to them afterward -- ugly)
> > >
> > > * second solution: replace foonotes by captionless floats: this way
> > > refs and defs can easily be separated. Example::
> >
> > the problem as i understand is you want to jump from/to the footnote.
> > i would use footnote (at least for now) aslong as footnotes are on the
> > same page i donot care to jump.
>
> The problem is not with jumping (aka hyperlinks). It's that if you want to
> use ``\footnote``, you have to give the note's contents at the moment you
> encounter the reference, i.e. when you visit the ``footnote_reference``
> node.
>
> But the contents are only defined once you've visited the ``footnote``
> node, which generelly comes *after* the reference!

Can't you search the DOM for the footnote body, based on the reference? If 
not, can you maybe have a simple transform that runs pre-writer that replaces 
footnote ref/footnote with a single footnote node?


     Richard

Hi.

In the project I'm using docutils in, we've decided to convert the ReST
sources to XML (with the docutils DTD) and then use some XSLT tools to
transform this into different formats. [#]_

However, we discovered that the output from "quicktest.py -r" is not valid
XML. There are two problems with the output:

1. The text contents are not encoded / decoded in any way; everything is just
   output verbatim. This is a problem if the input file is ISO-8859-1 and XML
   is supposed to be UTF-8. Some --input-encoding and --output-encoding
   options would be nice.

2. The "xml:space" attribute is used incorrectly. In the output one can find
   "xml:space=1"; the only valid values from xml:space are "default" and
   "preserved".

What's the best way to fix these issues? Perhaps a dedicated XML writer is
necessary (I hope not...)?

.. [#] The reason for this, as opposed to writing a new docutils writer, is
       mainly because we have a developer that has a lot of experience with
       XML/XSLT and no experience with docutils or Python: we believe that 
       this aproach will give us results faster, basically.

---
Adam Chodorowski <ad...@ch...>

Om du skiter kulor har du magproblem. Ät mer fibrer som asbest och 
knäckebröd.
    -- Brad S / Datormagaizin

Hey list, it's me again.

On Sep 23, 2002, at 15:27, gr...@us...  was heard saying:

> you could send me your version for review ?

No problem.

> > * the problem comes from the fact that footnote refs and footnote bodies are
> >   separated in ReST, whereas in LaTeX, they're not.
> >
> > * my first solution doesn't work (defining footnote bodies in the document
> >   head, and referring to them afterward -- ugly)
> >
> > * second solution: replace foonotes by captionless floats: this way refs and
> >   defs can easily be separated. Example::
>
> the problem as i understand is you want to jump from/to the footnote.
> i would use footnote (at least for now) aslong as footnotes are on the same
> page i donot care to jump.

The problem is not with jumping (aka hyperlinks). It's that if you want to use
``\footnote``, you have to give the note's contents at the moment you
encounter the reference, i.e. when you visit the ``footnote_reference`` node.

But the contents are only defined once you've visited the ``footnote`` node,
which generelly comes *after* the reference!

Anyways, using ``\footnote`` is not an option, since in ReST, footnotes can be
multiply referenced, which is impossible in LaTeX if you use the ``\footnote``
command (AFAIK).

And, as I humbly want to state, the figure-footnote solution (as presented in
my previous mail) works, and allows the LaTeX writer to be fully ReST
compliant (as far as footnotes are concerned).

.. troll::
	"Donald, don't use footnotes in your books."
	-- Mrs. Donald E. Knuth, in 'The TeXbook, vol. 1'

Cheers,
-- 
Julien T. Letessier     email: co...@me...
ENSIMAG Student         web: http://www.mezis.net

On Mon, 23 Sep 2002, Julien T. Letessier wrote:

> On Sep 23, 2002, at 12:25, gr...@us...  was heard saying:
>
> > i will check in the modifications into my sandbox (if it worls today this day)
>
> Fine.
>
> > * web-site is not a docinfo item so it will not be handled in the docinfo
> >   section. this is not recognizable in html as the handling is the same
>
> > * in latex i might choose to put author and date under latex control.
>
> Probably better visually.
but looks different, the latex article layout

                              TITLE
                             author
                             date

  :contact: me@home                   :web-site: http:www

and contact is in the table for docinfo website is a list following the docinfo.

>
> BTW, I won't be able to complete my version of the Writer today -- I'm having
> problems with py Python installation... and with ReST footnotes.
>
you could send me your version for review ?
> Please give me your opinion:
>
> * the problem comes from the fact that footnote refs and footnote bodies are
>   separated in ReST, whereas in LaTeX, they're not.
>
> * my first solution doesn't work (defining footnote bodies in the document
>   head, and referring to them afterward -- ugly)
>
> * second solution: replace foonotes by captionless floats: this way refs and
>   defs can easily be separated. Example::
>
SNIP ---
> What's your opinion about this method?

the problem as i understand is you want to jump from/to the footnote.
i would use footnote (at least for now) aslong as footnotes are on the same
page i donot care to jump.

cheers
-- 
 BINGO: Reporting
 --- Engelbert Gruber -------+
 SSG Fintl,Gruber,Lassnig   /
 A6410 Telfs Untermarkt 9  /
 Tel. ++43-5262-64727 ----+

I've been rather quiet on the group for a while, mainly because I've
actually had some rather satisfying work to do *at work* (allowing me to
scratch my "programming itch" whilst getting paid).

Anyway, it's become obvious to me that I'm not going to get around to
doing any significant work on pysource (you can find it in the sandbox,
under tibs), so I'd like to offer it up for others to take over/work
with. I reckon it could do with a solid refactoring, so anyone adopting
it shouldn't be afraid to start again from scratch using what was there
as one source of ideas. It's also worth noting David's Pysource document
in the main docutils release.

Meanwhile, I've got a paper on reStructuredText to write for the
ACCU/UKPython meeting next April (a different sort of fun)...

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"How fleeting are all human passions compared with the massive
continuity of ducks." - Dorothy L. Sayers, "Gaudy Night"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

Hi list,

On Sep 23, 2002, at 12:25, gr...@us...  was heard saying:

> i will check in the modifications into my sandbox (if it worls today this day)

Fine.

> * web-site is not a docinfo item so it will not be handled in the docinfo
>   section. this is not recognizable in html as the handling is the same

> * in latex i might choose to put author and date under latex control.

Probably better visually.

BTW, I won't be able to complete my version of the Writer today -- I'm having
problems with py Python installation... and with ReST footnotes.

Please give me your opinion:

* the problem comes from the fact that footnote refs and footnote bodies are
  separated in ReST, whereas in LaTeX, they're not.

* my first solution doesn't work (defining footnote bodies in the document
  head, and referring to them afterward -- ugly)

* second solution: replace foonotes by captionless floats: this way refs and
  defs can easily be separated. Example::

	my note [1]_

	.. [1] note body.

  becomes (without backrefs)::

	my note [\href{#note-1}{1}]

	\begin{figure}{b} % at least put the notes at the bottom...
		\hypertarget{note-1}
		\footnotesize
		[1] note body.
	\end{figure}

This isn't very LaTeX-ish, but it works, and is easy to realize.
Footnote ordering will be the order of the defs, not of the refs, which
probably is unimportant.

What's your opinion about this method?

BTW, Handling bibliography references/definitons is, of course, easier, since
they're pretty much the same (struture-wise) in ReST and LaTeX.

.. troll:
	Next problem: tables. Sounds hard, hmm? Well, I'm pushing stuff in
	and popping stuff off the self.context stack quite a lot... hoping
	it works.

	And, surprisingly, it does!

Cheers,
-- 
Julien T. Letessier     email: co...@me...
ENSIMAG Student         web: http://www.mezis.net

> > Engelbert, are you planning on continuing development of the LaTeX
> > writer or the PDF writer in the near future?  Or would it be OK if
> > Julien or someone else took over?
actually i currently have a project where docutils with a latex
writer might help me, so i will continue.

rlpdf is on hold for several reasons
* no one needs it currently
* pdflatex should do it and would do it better as developing the
  linebreaking in python/ps seams to be heavy.
* ..

-- 
 BINGO: Remarkable bird, the Norwegian Blue.
 --- Engelbert Gruber -------+
 SSG Fintl,Gruber,Lassnig   /
 A6410 Telfs Untermarkt 9  /
 Tel. ++43-5262-64727 ----+

On Thu, 19 Sep 2002, David Goodger wrote:

> Julien T. Letessier wrote:
> > I decided that I loved reST a few days ago, and that I'd love it
> > even more if it could output LaTeX. So I found this preliminary
> > LaTeX writer file in Engelbert Gruber's sandbox directory, and I
> > improved it (that is, made it a bit more functionnal).
>
> Julien, merci!
>
> Engelbert, are you planning on continuing development of the LaTeX
> writer or the PDF writer in the near future?  Or would it be OK if
> Julien or someone else took over?
i am willing to continue, but i have no time for theoretical work
means the writer is finished as sson as it fullfills what is needed

and i can see some problem s coming up,
interesting for all the project, because this would be the
second writer, but bigger ones, because the first writer and
the whole thing is targeted at html.

html for instances has no page length and no pagewidth, latex needs
this.

if it is ok for julien he could use my sandbox too
, if he needs his own for this, this is ok too

>
> Engelbert and Julien, would either or both of you be willing to
> complete the writer?  "Complete" doesn't mean it has to be perfect,
> just that it handles all Docutils constructs, as the current
> html4css1.py does.  Once complete, I'd like to put it into the main
> source tree.  Try running it over tools/test.txt to see where the
> issues are.
>
> I've been maintaining the HTML writer.  Once the LaTeX writer is in
> place, it will be maintained as well, hopefully with the assistance of
> people with more TeX knowledge than I.

>
>

-- 
 BINGO: Berichten Sie gegen den Projektplan
 --- Engelbert Gruber -------+
 SSG Fintl,Gruber,Lassnig   /
 A6410 Telfs Untermarkt 9  /
 Tel. ++43-5262-64727 ----+

Dethe Elza wrote:
> But reST actually converts to the DOM when you call asdom(), right?

Yes; what else should it do?

> So it's a potentially expensive operation.

Theoretically it's O(n), and you should only have to do it once.

>> Please don't think of *any* part of the current Docutils
>> implementation as written in stone.  It's all experimental, all
>> subject to change if we discover it's broken or deficient in some way.
>> That's what the "0." part of the "0.2" ("0.2.3", currently) version
>> number is meant to imply.
> 
> Point taken, and I'm glad we're able to have these discussions in that
> light.

I've found that recognizing that all code is disposable has been a great
liberator.  Someone said that any decent system needs to be written three
times: once to discover the issues, again to discover the *real* issues by
trying to solve the initial ones, and a third time to write an elegant,
general solution that really works.

> Some things jump out at me because I'm new to the project. If I find they're
> important enough to me, I may jump in and fix them.

Please feel free!

> Why is node.nodeName == 'Body' more expensive than isinstance(node, Body)? If
> it's because of a string compare rather than a pointer compare, then the
> strings can be interned and it's a pointer comparison again.  Or is there some
> other reason?

I neglected to point out that ``nodes.Body`` is an abstract superclass of
*many* concrete node classes.  There are many such "element category"
superclasses, roughly corresponding to those listed in
http://docutils.sf.net/spec/doctree.html#element-hierarchy.  The difference
is between ``isinstance(node, nodes.Body)`` and ``node.tagName in
('paragraph', 'bullet_list', 'enumerated_list', 'definition_list', ...)``.
Multiply by the maintenance required every time a new element class is
added, and the difference should be clear.

> But once I'm done with the include directives I want to take a look at a reST
> DOM -> reST text writer to get a feel for how tricky it will be.

One thing to remember: it's not a "reST DOM", it's a "Docutils DOM" (or
"Docutils doc tree", as I usually refer to it to avoid misinterpretation).
The doc tree (internal document representation) is independent of
reStructuredText.

>> In addition, there's a lot going on behind the scenes that the
>> Docutils DTD doesn't expose.  Try running ``html.py --dump-internals
>> input.txt ...`` to see what I mean.  ("--dump-internals" is an
>> internal, hidden option, for debugging.)
> 
> OK, but without knowing more about *why* that is, it looks like a bug
> to me.

It's mostly bookkeeping.  In the source text, we may see::

    Here is a reference_ to a web site.

    .. _reference: http://www.example.com/

Once parsed, the target URL has to be moved over to the "reference" text.
The internal data structures keep track of stuff like this.  Any browser or
other "user agent" has to do the same: create an internal database of
references and targets in order to make hyperlinks work.  That internal
database is merely an implementation detail.

> It's Tim Peter's koan, "explicit is better than implicit."  Once we start
> getting too much magic going on under the covers, python starts veering
> towards perl.  IMHO that's one of the major problems in Zope, and makes work
> in Zope, beyond the very trivial, much harder to do and to understand than it
> would otherwise be.

I don't think that's a valid comparison, but perhaps we're talking at
cross-purposes here.  There's nothing wrong with not exposing implementation
details where they're not relevant, such as in the output XML.  However,
these details do need more documentation.  I've tried to document them in
docstrings, but until there's a docstring extraction system in place you
have to read the source.

> One of my upcoming projects is to pull one of my unfinished novels out
> and serialize it on my weblog using reST.  I really like the way I can
> more or less forget about the tool and focus on writing--it's almost as
> good as a typewriter that way!

I had occasion on Friday to write a document unrelated to Docutils.  I used
reStructuredText in Emacs, and the text just flowed.  I was quite pleased by
how easy it was to write the words *and* markup out without having the
markup get in the way.  It's good to hear that I'm not the only one.  I'm
biased, so I can't trust my own good experiences without corroboration.

> Of course, a typewriter never gives you compile errors.  That's a major
> problem with XML and currently with reST.  I have little hope of seeing
> it solved in XML, but I think we can and should make an effort in reST
> to make errors rarer, clearer, and more easily found/fixed.

I agree completely.  In fact, I'm currently working on improving the
Reporter system to always report line numbers, and to report to stderr in
the GNU Tools format: "file:lineno: message".  Should be checked in soon.

> A casual user should never have to see a python stack trace, for instance.

Again, agreed.  The only time a stack trace should occur at present is with
a "SEVERE" (level-4) system message.  I suppose even that ought to be
suppressed though; just output the system message plus a line saying
"Processing stopped due to the problems reported above" or some such.

If there are any other stack traces happening, they're bugs and should be
reported and squashed.  I/O error handling could use some improvement; if
you specify a nonexistent file, you'll see a stack trace.

Another consequence of a "0.x" version: a bit of roughness around the edges.

> Fortunately, what's there is rich enough and stable enough to think
> about things like catching all the exceptions.  And it may only be that
> I see a lot of exceptions because I'm working on a) very long and
> complex documents, and b) actively changing the guts of the system.

(b) I can understand, but (a) shouldn't cause any stack traces (except
perhaps a MemoryError if the document is *that* long!).

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

>> I didn't know about asdom(), I'll have to explore that to see how
>> expensive it is and which DOM implementation it uses.
>
> It uses xml.dom.minidom, but it's intended to be able to use any DOM
> implementation (it's parameterized).

Excellent.  But reST actually converts to the DOM when you call 
asdom(), right? So it's a potentially expensive operation.

> Please don't think of *any* part of the current Docutils
> implementation as written in stone.  It's all experimental, all
> subject to change if we discover it's broken or deficient in some way.
> That's what the "0." part of the "0.2" ("0.2.3", currently) version
> number is meant to imply.

Point taken, and I'm glad we're able to have these discussions in that 
light.

>> I still think that the internals could be simplified and that this
>> would encourage more participation in the project.
>
> There are hairy parts of the parser code which could use refactoring,
> true.  You're working on one of them: the directive parser.  In the
> parser's core code (states.py), there's some duplicate code that could
> be refactored.  But that's low priority on my list at present.  It
> ain't broke (much), so it's not important (to me) to fix it (yet).

No, of course not.  Some things jump out at me because I'm new to the 
project.
If I find they're important enough to me, I may jump in and fix them.

>> I agree that the parser is the most complex component of reST, so if
>> it focuses on the parser and reuses architecture from the python
>> libraries for the rest of reST it may be easier to grok for a
>> programmer coming to it fresh.
>
> Ideally yes, but in practise it's too expensive.  It's just too
> convenient for a transform to say ``isinstance(node, Body)``, since
> the element hierarchy is built in to nodes.py.  This would be painful
> if we were using a DOM tree.

Why is node.nodeName == 'Body' more expensive than isinstance(node, 
Body)?
If it's because of a string compare rather than a pointer compare, then 
the strings can be interned and it's a pointer comparison again.  Or is 
there some other reason?


>>  Ideally I also want a writer to create reST from the reST
>> DOM.
>
> That last sentence seems to me the key, and that's exactly what's
> missing.  Transforming canonical Docutils DOM to internal nodes.py
> doctree should be reasonably easy, but without the reStructuredText
> writer, it's practically useless.  If I were writing a converter from
> XHTML or DocBook to reStructuredText, I would do it in a quick & dirty
> way, completely independently of Docutils (as I've described before).

Yes, and that's what I've done so far.  But once I'm done with the 
include
directives I want to take a look at a reST DOM -> reST text writer to 
get a
feel for how tricky it will be.

> In addition, there's a lot going on behind the scenes that the
> Docutils DTD doesn't expose.  Try running ``html.py --dump-internals
> input.txt ...`` to see what I mean.  ("--dump-internals" is an
> internal, hidden option, for debugging.)

OK, but without knowing more about *why* that is, it looks like a bug 
to me.
It's Tim Peter's koan, "explicit is better than implicit."  Once we 
start getting too much magic going on under the covers, python starts 
veering towards perl.  IMHO that's one
of the major problems in Zope, and makes work in Zope, beyond the very 
trivial, much
harder to do and to understand than it would otherwise be.

>  I don't know of a decent/standard DTD for novels out there.

Hah! That's perhaps the reason I like reST so much.  Right now I'm 
using it for technical documentation at work, but the reason I got into 
the work I'm in is because I was so frustrated with word processors and 
proprietary formats that I've put my creative writing on hold for 
several years to work on tools.

One of my upcoming projects is to pull one of my unfinished novels out 
and serialize it on my weblog using reST.  I really like the way I can 
more or less forget about the tool and focus on writing--it's almost as 
good as a typewriter that way!

Of course, a typewriter never gives you compile errors.  That's a major 
problem with XML and currently with reST.  I have little hope of seeing 
it solved in XML, but I think we can and should make an effort in reST 
to make errors rarer, clearer, and more easily found/fixed.  A casual 
user should never have to see a python stack trace, for instance.

Fortunately, what's there is rich enough and stable enough to think 
about things like catching all the exceptions.  And it may only be that 
I see a lot of exceptions because I'm working on a) very long and 
complex documents, and b) actively changing the guts of the system.

TTFN

--Dethe

>> Next I'm going to reimplement the include and raw directives using
>> the new function.  I'll then add examples to the HOWTO, along with
>> Aahz's corrections (thanks, Aahz!).
>
> I see you've added some more today.  Please let us know when you're
> done with the reimplementation and you're satisfied with the code.
> Then the integration will begin!

I'll do that.  I have include working (locally, I was on the train) and 
I need to investigate why raw isn't working.  I'm also dreaming up some 
new directives, now that they're easier to do.

--Dethe

Aahz wrote:
> !!!  Ah, now the lightbulb strikes about why I was having such difficulty
> understanding things earlier -- I kept looking for the transformer and
> couldn't find it.  Is there any place all this stuff is documented like
> this?

PEP 258 has a high-level overview of the Docutils design.  The diagram shows
that the transforms are run by the Reader and Writer, and the text
reinforces this.  My idea is to disconnect the transforms from the Reader &
Writer, add a new class in-between them (perhaps "Transformer"), and hang
the transforms off of that.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Dethe Elza wrote:
> I didn't know about asdom(), I'll have to explore that to see how
> expensive it is and which DOM implementation it uses.

It uses xml.dom.minidom, but it's intended to be able to use any DOM
implementation (it's parameterized).  I haven't actually tried it with
any other DOM, so I don't know if it works or not.  It's a case of
"don't add functionality until it's needed".  So if it's needed, it
can be added.  (By those who need it, of course!)

Please don't think of *any* part of the current Docutils
implementation as written in stone.  It's all experimental, all
subject to change if we discover it's broken or deficient in some way.
That's what the "0." part of the "0.2" ("0.2.3", currently) version
number is meant to imply.

> I still think that the internals could be simplified and that this
> would encourage more participation in the project.

There are hairy parts of the parser code which could use refactoring,
true.  You're working on one of them: the directive parser.  In the
parser's core code (states.py), there's some duplicate code that could
be refactored.  But that's low priority on my list at present.  It
ain't broke (much), so it's not important (to me) to fix it (yet).

> I've seen some complaints about the complexity of reST in toto,
> which I think could be addressed by modularizing reST, but that's
> another issue.

It's just people mistaking richness for complexity, and the ubiquitous
"I don't need it, therefore it's superfluous" crap.  It's mostly
bogus.  We can modularize the parser on a case-by-case basis, but I
think a general modularization will only increase the perceived
complexity.  Some modularization is already in place: the code behind
the "--pep-references" and "--rfc-references" options was ripped out
of the PEP reader, so that it could be used any time.

> I agree that the parser is the most complex component of reST, so if
> it focuses on the parser and reuses architecture from the python
> libraries for the rest of reST it may be easier to grok for a
> programmer coming to it fresh.

Ideally yes, but in practise it's too expensive.  It's just too
convenient for a transform to say ``isinstance(node, Body)``, since
the element hierarchy is built in to nodes.py.  This would be painful
if we were using a DOM tree.

> I thought a version of PyXML was part of the core now, but not
> 4Suite.

There's xml.dom (.minidom, .pulldom), xml.sax, and xml.parsers.expat.
That's it.  Your build or distribution of Python may include it, but
it's not part of the core.

> Besides, Optik is not part of core python, but it drastically
> simplifies the reST code, so it's included in docutils.

Optik *will* be part of core Python, as of 2.3.  And it's small enough
that we can include it with Docutils now.  One of the goals is for
Docutils itself to be included in the core stdlib, so we can only use
modules already part of the stdlib or scheduled to be included.  We
can't presume to get PyXML into the stdlib riding on Docutils'
coattails.

>>> And my *other* project of converting existing HTML and DocBook
>>> documents into reST for maintenance would be that much easier!
>> 
>> I don't follow this at all.  Can you elaborate?
> 
> Sorry, that wasn't very clear.  I want to think of the reST DOM as
> it's canonical form, so I can transform XHTML and DocBook to reST
> via XSLT.  Ideally I also want a writer to create reST from the reST
> DOM.

That last sentence seems to me the key, and that's exactly what's
missing.  Transforming canonical Docutils DOM to internal nodes.py
doctree should be reasonably easy, but without the reStructuredText
writer, it's practically useless.  If I were writing a converter from
XHTML or DocBook to reStructuredText, I would do it in a quick & dirty
way, completely independently of Docutils (as I've described before).

In addition, there's a lot going on behind the scenes that the
Docutils DTD doesn't expose.  Try running ``html.py --dump-internals
input.txt ...`` to see what I mean.  ("--dump-internals" is an
internal, hidden option, for debugging.)

> Thanks, that's a good start.  DocBook has added support for
> describing EBNF in documentation, as well as including modules for
> MathML and SVG, it is essentially a superset of XmlSpec, which is
> the *other* widely used XML documentation format (at least in the
> W3C).  I just had a wild idea that instead of inventing a new XML
> DTD for internal structure, reST could use DocBook (or a subset of
> it) for it's DOM representation. Like I said, a wild idea.

I looked at DocBook, XMLSpec, TEI, and HTML (and I've worked
extensively with DocBook and TEI in the past), but none of them fit.
I've borrowed ideas from some of them.  It was easier to start fresh
than try to shoehorn an existing system into use in Docutils.

>> For example, I know of no DTD that has the equivalent of a
>> "transition" element, although they're quite common in novels and
>> articles.
> 
> <hr /> doesn't qualify?

That's true, that's the one example I do know of.  But HTML doesn't
really count; I think of it as an output format, not as descriptive
markup.  I'm still puzzled why there's no transition equivalent in
TEI, which is a DTD for publishers!  They probably just use an empty
paragraph or some other kludge.  I don't know of a decent/standard DTD
for novels out there.

> Thanks for the eloquent feedback.

Any time.  Thank *you* for the penetrating questions.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Hi,

[...]
> > - as there is probably none of above: do you think it is possible to
> > port rST to php? more specifically:
> >   . is there any general experience in porting python to php?
> 
> I'm sure there's some out there, you just need to find it :) Seriously, I've 
> never seen any conversion information between the two. I've seen some 
> skin-deep comparisons, and they look similar - on the surface, PHP is Python 
> with added punctuation ... "and bugs" according to our PHP developer :)
[...]

I've only heard of projects working on the opposite direction.  For example at 
	http://www.zope.org/Members/mjablonski/PHParser
you can find a tool developed to port existing PHP scripts to Zope.

Personally I have never used any PHP.  But from everybody I've heard of
and who knows the languages PHP or Perl and Python, they have started to
port or rewrite their PHP or Perl stuff from PHP or Perl to Python.

Most web servers nowawdays run on Linux and Linux systems usually
come with Python preinstalled (the major distributor Red Hat uses Python 
for its installer 'anaconda' and the future Linux kernel configuration 
manager is also written in Python). 

Maybe it would be possible for you to embed the fine Python implementation 
of reST written by David Goodger into your PHP application with some 
glue code?

Porting Davids implementation to another language looks like a lot of work.

Regards, Peter
-- 
Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany
office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen, Germany)

On Fri, Sep 20, 2002, David Goodger wrote:
> Richard Jones wrote:
>>
>> See PEP 258 for a description of the processing framework itself.
>> That works along the basic lines of:
>> 
>> 1. read from a source document, parsing the structure out of it and
>>    creating a DOM tree
>>    (http://docutils.sourceforge.net/spec/doctree.html)
>> 2. optionally do some transforms on the tree (NOOP for current ReST
>>    reader)
> 
> Not precisely true.  In fact, there are several transforms that are
> done before the doc tree is handed off to the writer: doc title,
> docinfo, references, footnotes, substitutions, etc.
> 
> However, please note that the present method of applying transforms is
> due for an overhaul.  I'm thinking of moving the responsibility for
> running the transforms away from the Reader & Writer and into a new
> intermediate class, perhaps just a super-Transform.  Readers, Writers,
> and other objects will register the transforms they want applied, into
> a list sorted by priority.  The transforms will be applied to the doc
> tree in order.  Transforms may insert further transforms into the list
> dynamically (which would be re-sorted).  Transforms that are now
> associated with the Reader will simply have higher ("to be run
> earlier") priority.

!!!  Ah, now the lightbulb strikes about why I was having such difficulty
understanding things earlier -- I kept looking for the transformer and
couldn't find it.  Is there any place all this stuff is documented like
this?

(Now I understand how better how my index entries need to be handled.)
-- 
Aahz (aa...@py...)           <*>         http://www.pythoncraft.com/

Project Vote Smart: http://www.vote-smart.org/

On Friday, September 20, 2002, at 07:55 PM, David Goodger wrote:

> Dethe Elza wrote:
>> Also, if we parse directly to a DOM it makes reST more flexible and
>> easier to port, since a DOM binding exists for most languages.  Many
>> DOMs use either SAX or Expat to build the DOM itself, my idea would
>> be to replace the low-level parser with reST.
>
> I don't see how that would improve flexibility.  The parser can
> already build a real DOM tree; just call ``document.asdom()``.  What
> benefits would a DOM approach provide?  I'm not being defensive; I'd
> really like to know.  If there is a benefit that outweighs the cost,
> it should be explored.

I didn't know about asdom(), I'll have to explore that to see how 
expensive it is and which DOM implementation it uses.

>> Building up a true XML DOM internally has several advantages.  More
>> potential developers would be familiar with the API than are
>> currently comfortable with the reST internals.
>
> I don't think the document tree is the bottleneck.  Rather, I think
> it's the complexity of the parser.  Unfortunately, parsing
> reStructuredText *is* complex, because it has to grok two-dimensional
> patterns that humans understand implicitly.  It's the curse of
> user-friendliness. ;-)

Yes, that's certainly true, and one of the things I really like about 
reST is the
effort it takes to make the document author's life easier.  I still 
think that the
internals could be simplified and that this would encourage more 
participation
in the project.

I've seen some complaints about the complexity of reST in toto, which I 
think could be addressed by modularizing reST, but that's another 
issue.  I agree that the parser is the most complex component of reST, 
so if it focuses on the parser and reuses architecture
from the python libraries for the rest of reST it may be easier to grok 
for a programmer
coming to it fresh.

>> Writers could be written in XSLT without knowing anything about reST
>> besides it's DTD.
>
> This can already be done: just use ``document.asdom()`` then run that
> through the XSLT engine.  The reason we don't go that route is because
> there is no XSLT engine in core Python.  If PyXML is ever incorporated
> into the core, we can re-examine that decision.

I thought a version of PyXML was part of the core now, but not 4Suite.
Besides, Optik is not part of core python, but it drastically 
simplifies the reST code, so it's included in docutils.

>> And my *other* project of converting existing HTML and DocBook
>> documents into reST for maintenance would be that much easier!
>
> I don't follow this at all.  Can you elaborate?

Sorry, that wasn't very clear.  I want to think of the reST DOM as it's 
canonical form,
so I can transform XHTML and DocBook to reST via XSLT.  Ideally I also 
want a writer to create reST from the reST DOM.

>> Even further off-topic, the docs mention that reST has constructs
>> which are missing from DocBook.  What are they?
>
> There are plenty.  Off the top of my head: field lists, option lists,
> decorations (headers & footers), doctest blocks, line blocks,
> transitions.  None of these are difficult to render or approximate
> using regular DocBook elements, it's just that there's no one-to-one
> correspondence.  Even in elements where there *is* a strong
> correspondence, some are not completely compatible, such as definition
> lists.  It is the goal of http://docutils.sf.net/spec/doctree.html to
> document all of this; any assistance would be gratefully accepted and
> much appreciated.

Thanks, that's a good start.  DocBook has added support for describing 
EBNF in documentation, as well as including modules for MathML and SVG, 
it is essentially a superset of XmlSpec, which is the *other* widely 
used XML documentation format (at least in the W3C).  I just had a wild 
idea that instead of inventing a new XML DTD for internal structure, 
reST could use DocBook (or a subset of it) for it's DOM representation.
Like I said, a wild idea.

> The Docutils document model was designed by me (with much input, of
> course), as it makes sense to me.  I've had some experience with
> various models, including DocBook and TEI, and I've designed several
> DTDs before.  Every document designer has different sensibilities, so
> differences and incompatibilies are inevitable.  For example, I know
> of no DTD that has the equivalent of a "transition" element, although
> they're quite common in novels and articles.

<hr /> doesn't qualify?

Thanks for the eloquent feedback.

--Dethe

Richard Jones wrote:
> See PEP 258 for a description of the processing framework itself.
> That works along the basic lines of:
> 
> 1. read from a source document, parsing the structure out of it and
>    creating a DOM tree
>    (http://docutils.sourceforge.net/spec/doctree.html)
> 2. optionally do some transforms on the tree (NOOP for current ReST
>    reader)

Not precisely true.  In fact, there are several transforms that are
done before the doc tree is handed off to the writer: doc title,
docinfo, references, footnotes, substitutions, etc.

However, please note that the present method of applying transforms is
due for an overhaul.  I'm thinking of moving the responsibility for
running the transforms away from the Reader & Writer and into a new
intermediate class, perhaps just a super-Transform.  Readers, Writers,
and other objects will register the transforms they want applied, into
a list sorted by priority.  The transforms will be applied to the doc
tree in order.  Transforms may insert further transforms into the list
dynamically (which would be re-sorted).  Transforms that are now
associated with the Reader will simply have higher ("to be run
earlier") priority.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Dethe Elza wrote:
> Sorry for the delay.

No need for apology; beggars, choosers, and gift horses' mouths
obviate the need.

> However, I have finally gotten my sandbox build on SourceForge and
> in it is a copy of parsers/rst/directives/__init__.py which I
> believe approximates the parse_directive function that we've
> discussed here.  Please feel free to look at it and suggest
> improvements and point out problems.

Thanks!  Will do.

> Next I'm going to reimplement the include and raw directives using
> the new function.  I'll then add examples to the HOWTO, along with
> Aahz's corrections (thanks, Aahz!).

I see you've added some more today.  Please let us know when you're
done with the reimplementation and you're satisfied with the code.
Then the integration will begin!

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/