[Tinyos-devel] TEP community review process

[Neil Hancock]

(2nd attempt - the first one got held for review 'cause it was over 40KB)
Hi Phil

Thanks for the detailed response. 

As an analogy to another professional field - I would like to suggest you
don't see buildings going up and then detailed plans being drawn up
afterwards.

The target goal of software engineering as a discipline is to define what
needs to be done beforehand - the architectural plans - and then
implementing it - is a highly desirable (if somewhat elusive) goal for
software engineering. 

However, the complexity of software and the unknown number of unknown
unknowns seems to make larger software projects unpredictable as large
numbers of expensive high profile project failures have demonstrated. 

The waterfall project implementations have been much more successful. That
is define one set of reasonable goals (TinyOS 1.x) work towards them, and
then with experience and knowledge under the belt, define another set of
goals (T2.x)

Of course discussions like this can go on forever, and each project has to
take its own cut at what are reasonable goals for the next phase of
implementation.

I would like to suggest that the TEP community review should be more to
promote an open listening process - what needs might the TEP meet, and what
might be missing from the TEP.

 As you've identified the "peer" review by the core working groups happen
earlier in the project cycle when all the major decisions are made and a cut
at the scope of the TEP is probably verbally discussed (and not written
down). The unanswered question is then;  how well will it work and how
useful will it be. 

Would it be more simple to view the TEP community review discussion as part
of building the community "customer" relationship.

That is- early staging publishing the TEP (it exists!!!), listening (anybody
who can understand it, what da ya think of it?) and documenting the response
and publishing it with any adjustments that were made. If it becomes
apparent that input is being effective, that will make it a very successful
part of the TEP review process.

Subtle, but different from the "peer" working group review. 

The reality is that one has to say what can be delivered with what available
resources. Other information collected has the potential to become input
into the next "waterfall" pass, whatever form that takes.

In a past life a very successful strategy that worked for me on one project
- the marketing guy wanting everything he could think the end customer would
ever need. It was about a monitoring system connecting a cabinet of LeadAcid
batteries to a TCP/IP stack (forefront of early 90's tech), and have a push
button user interface that could be work with gloved hands in winter in
Alaska. 

I defined the functionality that I estimated I could meet in the time frame
that was defined. In a project review, he was mad as hell because he didn't
get everything, (and my management didn't put more people of the project). 

I delivered on the functionality I defined in the time frame I had been
given, for what was considered a successful engineered project.  If I hadn't
put a stake in the ground on functionality it could have become an
unsuccessful project J

More responses below

Neil


> -----Original Message-----
> From: Philip Levis [mailto:pal at cs.stanford.edu]
> Sent: Tuesday, November 07, 2006 11:22 PM
> To: Neil Hancock
> Cc: 'Kevin Klues'; 'Matt Welsh'; 'TinyOS-Devel list'
> Subject: Re: [Tinyos-devel] Request for comments: TEP-108
> 
> 
> On Nov 7, 2006, at 9:24 AM, Neil Hancock wrote:
> 
> > To the TinyOS 2.0 core group
> >
> > Congratulations to the  TinyOS 2.0 working groups. A truly amazing
> > achievement to get the next level of functionality released, and
> > hopefully
> > working. I don't see any pointers to test data in the release
> > notice - but
> > hopefully everybody has done their best and had fun with it.
> >
> > However it does raise a question that I asked earlier and I feel I
> > should
> > ask again:
> >
> >      What is the purpose of the Tinyos-devel TEP community review
> > requests?
> >
> > It seems to me that the TEP community reviews are happening at a
> > late stage
> > in the project cycle.
> >
> > I initially volunteered to review TEP-108, and haven't done it due
> > to being
> > snowed under in other areas.
> > I have reviewed other TEPs and my sense of the TEP review's that I
> > offered
> > was that even with some really basic comments, the TEP review is so
> > late in
> > the project cycle that the authors don't feel able to incorporate
> > anything
> > more.
> >
> > I don't propose to respond to TEP review requests unless it is
> > clear that
> > the TEP review is early in the project cycle, that the TEP hasn't
> > already
> > been implemented and there is a clear intent to seek community
> > input to
> > shape the TEP.
> >
> > It could also be a failing in the TEP life cycle. That the
> > documents are
> > "dead" documents and can't evolve once they are release.
> > Another possibility in the TEP life cycle is that the TEPS are living
> > documents, with multiple reviews and releases and define phases, so
> > that the
> > implementation can be phased.
> >
> > Keep up the good work - it is virtually impossible to do everything
> > right
> > straight away, so trial-and-error has to be built in.
> 
> 
> Neil,
> 
> These are good points. Thanks for bringing them up.
> 
> You're definitely right that many of the TEPs emerging now are
> relatively late in their development cycle. 

[nh:] 
So this is a problem if you want the community review to be taken seriously.

It's commonly the case
> that a given TEP has gone through 2-3 rewrites (and re-
> implementations) before being sent to the Steering Committee for
> community review. In that way they are living documents, which have
> been in the repository, whose discussion has been on the publicly
> accessible mailing lists, and whose re-implementations have been in
> the repository.
> 
> We tried sending a few earlier (e.g., 106 in March, 101, 106, 107,
> 109 in July 2005). There were some comments, which were incorporated
> into the documents. TEP 103, for example, had significant feedback
> which was incorporated into the finalized document. If you look
> carefully, the interfaces in TEP 103 are different from those
> initially presented for community review.
> 
> Unfortunately, the idea that a Documentary TEP might be seriously
> considered *before* an implementation exists is a bit contrary to the
> intent of that class of TEP. There may not be a full implementation,
> but the general approach has been to at least have prototypes as well
> as prototype users, in order to assess whether the interfaces are
> feasible. Otherwise we get into a situation of lots of people
> suggesting changes whose unforeseen ramifications and difficulties
> others have to deal with.
> 
[nh:] 
This to me is inconsistent, and is partially the lack of definition with the
TEPs that is causing me to question the  TEP community review process. 

In my mind the Design Document (TEP) should have 1) the layer described 2)
interfaces defined and described. 

TEP107 & TEP108 being a great example of what is possible.

The author may, depending on the project have defined the basic framework of
the code modules outside the TEP, to be able to accurately put them in the
TEP.

I have found an edges in design method to be very successful, where high
risk algorithms can be tested in a proto-type framework to begin with, to
verify the essential nature of what needs to be documented in the design
document.

In an ideal world, implementation happens after design documentation - and
implementation is about completing the coding, simulating and testing.

If anything near a full implementation has been implemented, with
significant testing, then the implementation programmers (TEP author?) is
going to be still sweating from the exertion of programming, simulating and
testing AND going to be incredibly resistant to any suggestions of change to
their piece of artwork. 

For convenience as it's a gray area, and other figures might work - I would
suggest that 

a) an implementation phase has been started if more that 30% of code has
been defined and more than 20% of expected module testing effort has been
expended, and 

b) design documentation phase has been exited if greater than 80% of coding
and 50% of module coverage testing has been completed.

> Ultimately, the purpose of the TEP review process is for the authors
> to gain feedback from the community on their design decisions. The
> most extreme case of this is finding bugs and fundamental technical
> problems; lesser cases include unclear specifications and wordings.
> Ultimately, it's up to the shepherd and the authors which comments
> make their way into the TEP. If other developers feel that the TEP is
> unsalvageable and a completely different approach is needed, then
> it's always possible to write a TEP about a different approach.
> 
> TinyOS's component model is open to having multiple implementations;
> the implementations that are in the core are not intended to be the
> be-all and end-all of TinyOS code. The TEPs have (for the most part)
> been carefully written with this in mind.
[nh:] 
The nesC component model is a great building block, and the TEPs evolution
should demonstrate if it can make a difference with embedded software's
maintainability and complexity issues.

> 
> Phil