[Tinyos-devel] Request for comments on TEP 103, due August 10th

[David Moss]

I think we're headed in the right direction with your BlockStorage comments,
Johnathan.   From what I gather, BlockStorage is meant to be a sort of
general purpose way to access flash with the added benefit of data
reliability through CRC checking, right? 

My first version of blackbook, as you may recall, was built on top of
BlockStorage.  I got it working and tested on the STM25P implementation with
a few tweaks to make it work right:

1. Don't allow the app to write to the last page of any sector on the STM25P
- you said this was fixed in 2.x, which is great.

2. Blackbook needed access to read/write multiple sectors in flash and erase
individual sectors.  Maybe this doesn't fit the BlockStorage model very
well, because to make this happen on my end, I made each sector its own
volume.  When I wanted to erase the sector, I'd mount() and erase() it, then
mount() back to volume 0 to keep the read/write addresses constant.  Kind of
cumbersome, but it worked.  I'm not sure with the concept of volumes how to
make it any better unless I adjusted my own application.  Having a volume
that spans multiple erase units and then being able to erase parts of the
volume is what I don't think fits well with the BlockStorage model - it's
probably not the point of BlockStorage, even though that's how I was using
it.


Then I met up with software implementation incompatibility issues between
the two flash types:

Blackbook, upon boot, would have to read the flash to locate where it left
off - simply find the newest write location.  It would do this on a sector
basis first, i.e. check the beginning of sector X for data.  If it found
data at the beginning of the sector, it would dive into that sector and look
for where the data ended.  After doing this loop for all sectors, Blackbook
knew where the newest write page was located in each sector in flash.  Any
application may need to do this - maybe not to the degree of complexity that
Blackbook did - but finding the next available write location is always a
huge task.

My point of explaining all that is in the AT45DB implementation, the data
never got erased from any other page in flash other than the first page.  I
do understand that erasing the CRC's existing in the first page of the
AT45DB flash was enough to invalidate the rest of the data on flash when it
is explicitly checked, but the fact that the data was still physically there
elsewhere on flash became a huge problem because Blackbook didn't know where
to start writing.  Being initially tested on the STM25P, Blackbook would
mount to whatever volume and erase it, only to find that none of the data
got erased.  That's fine if that was how it had to be, but maybe an extra
command or two would have helped to automatically find the next available
write location.  

I guess this problem comes down to: how does a developer know what flash
address is ok read from and write to?  And if I'm writing data to various
addresses all over the flash, how is it possible to use CRC's to double
check if any data is valid?  Maybe the two flash types have different ways
of determining which address is good to read/write, but having the ability
to do that automatically would have helped a lot when trying to write an
application that uses BlockStorage.  I'm not sure how the implementations
are, but I don't think erasing the volume each time a new chunk of data
needs to be written is a good idea and not very efficient on time or
batteries.

I'll continue to actively think about solutions to some of the problems I
faced.  I think that LogStorage and ConfigStorage could probably be pretty
similar in behavior regardless of which flash type they're on, but the
former BlockStorage implementation seemed to just need a few software
modifications to make the two flash types act similar.  Adding in some
developer friendly commands, like the ability to locate the next available
write address automatically, would probably help maintain application layer
consistency.

Thanks Johnathan,
-David



 

-----Original Message-----
From: jonhui at gmail.com [mailto:jonhui at gmail.com] On Behalf Of Jonathan Hui
Sent: Wednesday, July 26, 2006 12:25 PM
To: David Moss
Cc: tinyos-devel at millennium.berkeley.edu; Ramesh Govindan; Joe Polastre;
handzisk at tkn.tu-berlin.de; David Gay
Subject: Re: [Tinyos-devel] Request for comments on TEP 103, due August 10th


On 7/26/06, David Moss <dmm at rincon.com> wrote:
> As a developer who's using this technology, time is one of my team's most
> valuable resources.  If my team has to spend twice the amount of time
> testing and developing code just to get it to run on two different
> platforms, then that's not worth it.  The fact that David Gay and
Johnathan
> are both writing the same end functionality doesn't seem worth it to me
> because their time is equally valuable.  Trying to get people to adopt
> hardware-dependant technology is much more difficult than getting people
to
> adopt hardware-independent technology.  If we eventually have 50 different
> flash types, who's going to write and test all that code for the basic
> storage implementations?  Not me, and not my team. Time to erase is one
> thing, but time to market is another.  An increase in nitty gritty
> bit-banging performance is usually not worth a delay in getting to a
robust
> and finished product.

Depends what your end goals are. If you're trying maximize the
performance a chip can give you while minimizing resource consumption,
then having a separate implementation per chip is desireable. Note
that you can do this while still providing a chip-independent
interface. We have had the luxury of using "fat" nodes since they are
what Crossbow, Moteiv, etc. provides. When cost is an issue, you might
want to spend the extra time to do the chip-dependent optimization
that still provides a chip-independent interface so that  your app
still works.

> Having optional platform-dependent and -independent interfaces is probably
> the best route to meet everyone's needs.  If you want your app to work
> everywhere, use the platform-independent interfaces.  If you want your app
> to take full advantage of extra features on a specific chip, then use the
> implementation tailored to that hardware.  Full documentation should be
> provided on how these interfaces work, how to use them, and what to
expect.

Don't get me wrong, I think application portability is important. It
really depends where you draw the "app" line. If most developers just
want a logging abstraction, then that's what they'll use and they
could care less if there's a common or different implementation for
each chip. You are different in that Blackbook is your "app" (the
product you're providing).

To support your concerns, we should look further into what the changes
should be. I'd be in favor of modifying the BlockStorage interfaces
since that seems pretty close to what you need (read/write/erase) and
add the erase unit gets. Then change commit() to sync() which commits
to stable storage. No need to differentiate between sync() and flush()
here. How does that sound?

> I guess what it comes down to is, I can jump in a hybrid or a ferrari and
> expect them to behave the same when I push the pedals, without having to
> reconfigure myself, the car, or know how their engines differ.

Just don't slam the gas pedal in a ferrari and expect to stay on the road
;-)

-- 
Jonathan W. Hui
jwhui at cs.berkeley.edu
http://www.cs.berkeley.edu/~jwhui/