Arbitrary Deadlines (Generalized Midnight) Spec

The background for this is in the announcement on the blog. Here’s the nitty gritty of what we’re (eventually) implementing! [1]

1   Showing the Deadline on the Goal Page

One of the tricky things about this change is that everyone is very used to midnight (or 3am) and so — even if you opt in to a different deadline — it has to be kept very front-and-center. Next to the countdown timer (somewhere in the box with it) will be shown the time of day of the deadline. When we first make this live, that time of day may need to be especially prominent. We do still want the countdown itself even though it’s redundant information because we want to convey that it’s dynamic and to have that ticking time bomb feel. But the time of day deadline could be big and the countdown small. (Before going live we could make the time of day small and keep the countdown big and then swap them as part of officially launching.)

2   Night Owls vs Earlybirds

How do we distinguish between someone who wants a 6am end-of-day because they want to be able to pull a beemergency all-nighter and someone who wants a 6am end-of-day because they want to be forced to dispatch their beemergencies before breakfast? They’re both choosing 6am but they mean quite different things.

It will be made obvious by picking the deadline via a slider:

[6am ---- 9am ---- 12pm ---- 3pm ---- 6pm ---- 9pm ---- 12am ---- 3am ---- 6am]

If you slide past midnight then a little footnote appears below the slider:

(Night-owl mode! After midnight so technically the next day, which is fine.)

If you slide before noon then the footnote is:

(Earlybird mode! Dispatch your beemergencies before breakfast.)

Webcopy for confirmation of a deadline before midnight (or maybe just an FAQ unless we can make it more concise than this):

WARNING: A deadline before midnight counts as the day boundary, even if it’s in the morning. If you manually add data for a given day then we respect the date you specified (though if it’s an emergency day and the graph has already derailed then it’s too late, the data’s accepted but won’t undo the derailment). If autodata comes in after the deadline then it’s displayed and plotted for the following day.

NB: To simplify the initial implementation we’re not going to allow earlybird deadlines before noon.

3   Changing the Deadline in Settings

A tricky thing about choosing your deadline is preventing you from pushing the deadline back as it approaches, defeating the whole point. Akrasia-proofing it would be fine if that’s what’s easiest to implement (we need a generalized akrasia-proofer for various fields anyway) but it’s harsher than necessary. But overharshness — having a suboptimal deadline for a week — is not the end of the world. Even if you had an earlybird deadline when you wanted a night-owl deadline, just remember to think of orange days as your beemergency days for a week. [2]

4   Daystamps on Datapoints

We treat the deadline as the day boundary, meaning that datapoints after midnight but before a night-owl deadline count as what’s technically the previous day. Similarly, but potentially more confusingly, datapoints after an earlybird deadline count as the following day. That may sound ugly but it doesn’t violate the sanctity of data to the point of actually changing timestamps. The fudging is just in how datapoints are displayed and plotted.

Autodata

If you have a night-owl deadline then datapoints timestamped after midnight but before the deadline are treated as if timestamped for the previous day. They’ll display that way in recent data and plot that way on the graph.

Technical Note: The deadline is stored as a number of seconds after midnight, which is a negative number for earlybird deadlines. For example, a night-owl deadline of 3am would be \(3\cdot60^2\), a standard midnight deadline would be 0, and an earlybird deadline of 11pm would be \(-1\cdot60^2 = -3600\). A reasonable range for deadlines is \(-18\cdot60^2\) to \(6\cdot60^2\) (6am earlybird to 6am night-owl).

Given a deadline, \(d\), defined like that, compute the daystamp (year-month-day string) for a datapoint \((t, v)\) by converting unixtime $$t - d$$ to year-month-day in the user’s timezone. The formula is general, applying to both earlybird and night-owl deadlines. For example, with an earlybird deadline of 9am and a datapoint on the 30th, the daystamp is the 30th until 9am, and the 31st — though that’s blatantly “tomorrow” — thereafter.

This is very much what people expect in night-owl mode (cf “it’s still the 30th to me!” and “I want to define when my day ends!”). It’s less expected in the earlybird case but at least it’s consistent. And, again, the timestamp is never altered, we just use \(t-d\) (per technical note above) for turning the timestamp into a daystamp (year-month-day string).

Mandata

Carets simply expand to Right Now (or shifted by a multiple of 86400 in the case of multiple carets) and thus work just like autodata datapoints. This fixes the longstanding confusion with having to submit a double caret after midnight.

Any datapoint given manually with an explicit day of month gets parsed to the dayfloored [TODO] timestamp (the earliest possible time on that day) and then has the deadline value added. Since all timestamps have the deadline value subtracted for the purposes of displaying and graphing, this ensures that a datapoint entered as, say, “the 30th” always displays and graphs as the 30th.

Scenario: You have an eep day on the 30th with a 9am earlybird deadline and you miss it. You derail and get rerailed. Then you add a datapoint of 30 1 which is timestamped as 9am on the 29th, right after yesterday’s deadline so is shown to the user as the 30th, as they expect. If the derailment wasn’t legit it could be undone and the data would be enough to keep you on track. That’s the same as the status quo where you forget to enter data in time but enter it retroactively.

Newbee Data Entry

The only weirdness here is that we say, e.g., “Today (30)”.

Case 1: For night owls the word “Today” gets this footnote during the night-owl zone (midnight till the deadline):

It’s after midnight but it still counts as the [30th] till the [3am] [deadline](link to settings).

Case 2: For night owls when you choose “Yesterday (29)” during the night-owl zone there’s no footnote (too convoluted to explain that it’s technically the day before yesterday and it’s not so critical to distinguish those anyway).

Case 3: For earlybirds during the earlybird zone, “Today (30)” turns into “Tomorrow (31)” and gets this footnote during the earlybird zone (deadline till midnight):

It’s after the [5pm] deadline so it counts as the [31st] now.

Case 4: For earlybirds, “Yesterday (29)” turns into “Today (30)” at the deadline and gets this footnote:

It’s after the [5pm] deadline so it counts as the [31st] now but you can still enter retroactive data.

5   Graph Refresh Schedule

Actual midnight is irrelevant. Refreshes happen when you hit refresh, before sending reminders (including zeno reminders), and at the deadline. (For some autodata sources, new data is pushed to Beeminder which also triggers a graph refresh; for others, Beeminder pulls new data when refreshing.)

Beebrain only knows about the daystamps so, for example, if you have datapoints today that are both before and after your earlybird deadline, then Beebrain will plot the graph with asof equal to tomorrow’s daystamp, which is consistent with thinking of datapoints after the deadline as counting for tomorrow.

Scenario: If your graph is currently orange and a refresh is happening now at the 9am earlybird deadline (or one second after) then the :asof will be tomorrow’s daystamp and the graph will refresh as red. And 24 hours later it will derail.

6   API Changes

We don’t bump the API version, we simply fix the unixtimes to be correct [4], and include an additional daystamp field (eg, “20140831”) along with the unixtimes. We straw-polled users of the API and the consensus was that we should just fix the timestamps in the API and not worry about backward compatibility with the previous craziness.

Technical note: We use daystamps like “20140831” instead of “2014-08-31” not just to shave off storage/bandwidth cost but to avoid confusion that might ensue if you left off preceding zeros, like “2014-8-31”. The condensed version is ascii-sortable as well as numerically sortable. And it avoids the question of what delimiters are allowed.

7   Transition

  1. Generalize the goal.brainby field
  2. Possibly bump the API version if we think people may have been depending on the above brokenness
  3. Can of worms with conveying dayfloored timestamps to Beebrain (also related to time zone kludginess)
  4. Always send :asof to Beebrain as a daystamp (alternatively, send :deadline and :timezone (tzinfo string) so Beebrain can compute the daystamp of Right Now)
  5. Countdown counts down to the deadline
  6. Zeno polling uses the deadline as the asymptote
  7. Newbee data form should show current day-per-deadline. E.g. at 10:01am on Aug 8 with a 9am earlybird deadline it should show “Today (09)”
  8. Store deadline (or daystamp) and tzoffset with datapoints
  9. Akrasia-proofing the deadline field
  10. Interface for setting deadlines (Does this need to be even fancier?)
  11. Datapoint parsing: carets expand to right now +/- n days
  12. Datapoint parsing: store actual urtext
  13. Accept real timestamp given for datapoint if constructed with an actual timestamp
  14. Do the correct thing with timestamp/daystamp for datapoints entered with [date] [value]
  15. Indicate that the deadline is akrasia-proofed (and include indication of transition if one is pending)
    — — — MENDOZA LINE: ABOVE NEEDED FOR BETA DEPLOYABILITY! — — —
  16. Range-check / limit on deadlines (e.g. force to be between 6am earlybird and 6am night owl)
  17. All existing goals get their official deadlines moved to 3am
    — — — MENDOZA LINE: ABOVE NEEDED FOR NON-BETA DEPLOYABILITY — — —
  18. Fix zeno polling / reminders (currently can’t get reminded for an earlybird goal before 3am day-of)
  19. TOD of deadline bigger in the countdown
  20. For API we probably want to let people pass the daystamp to create a datapoint
  21. Completely get rid of daysnapping in codebase (or are there places where it is still relevant?)
  22. Fully refactor away the horrifying old solution to the daystamping problem (newyorkified unixtimes)

8   Open Questions / Things to Reconsider Later

  1. Decide whether we’ve lowered the bar too much for weaseling since you can report data (specifying the day of the month) after the deadline and it’s adjusted back to just before the deadline. That means it will appear for the day you entered it even though you missed the deadline. (Autodata will show up as tomorrow if you’ve missed today’s deadline but mandata with day-of-month given explicitly won’t.) That could make it too easy to reply to the legit check like “hey, so that was totally close enough, right? can you just hit undo on that derailment real quick?”. Tentatively we’re just ok with that.
  2. The whole thing where a datapoint specified as “the 30th” can get an official timestamp that’s blatantly the 29th. Like we add the deadline value to store it and then subtract the deadline value again to display it. It seems weird but it’s the right thing for autodata where we store the true timestamp. It’s just in the case of mandata where we need to adjust it so that when the user says “the 30th” it always actually displays that way.
  3. When we sample Duolingo we don’t get the actual timestamp, just the current number of points. But that’s ok because we sample right before the deadline so we can treat the current number of points as timestamped Right Now.
  4. Alice’s ISO 8601 Proposal: We bump the API version and switch to giving all timestamps as ISO 8601 strings (eg, “2014-08-31T23:59:59-07:00”). Note that GitHub uses ISO 8601 dates, so that’s a very compelling precedent. The UTC offset would encode not the timezone (that’s a muddy mapping anyway) but the deadline. That way the first 10 characters of the date string would represent the daystamp that the datapoint counted for. (Or would it confusingly be that we need to have the offset go in the opposite direction of the deadline to have that property? The fact that it was hard to keep that sort of thing straight is part of why we decided to punt on ISO 8601 for now. Also: Unixtime is more convenient for developers and most API usage (like Exobrain, or things that just pass new data to Beeminder) doesn’t care about daystamps. We could accept incoming unixtime regardless but that’s inconvenient not to get out what you put in.)


 

See also: Continuous Beeminding.

Footnotes

[1] Crazy alternative proposal not currently on the table: There’s no such thing as “6am the next morning”. You can set your deadline to be 6am but that would mean bright and early on the emergency day. Night owls would then have to be aware that the real emergency day — when they can pull the all-nighter to get back to safety — is when they’re in the orange. With more flexible zeno polling and better use of the panic threshold that might be tenable. Right now everything is very eep-day-focused and it would be too confusing.

More flexible zeno polling means starting on the previous day. Otherwise a night owl who wanted, say, a 3am deadline would not be able to have Beeminder tell them to panic until 3 hours before derailment. But the generalization is actually pretty natural as well: You pick a time of day for zeno start time and if it’s after your deadline then it takes it to mean the previous day at that time. (That also means you could have zeno polling start up to 24 hours before the deadline. If the deadline is, say, 5pm, just make the zeno start time be 5:01pm.)

[2] But if it’s not too much of a special case, here’s an alternate spec for changing the deadline tomorrow:

Ideally any changes to the deadline simply take affect the next day. To pin that down, if the current deadline and the new deadline are both 12 hours [3] or more in the future then just allow the change immediately. If either the current or the new deadline are less than 12 hours in the future then still allow any change, but display a warning:

(The new deadline of XX:XX will take effect after the upcoming YY:YY deadline.)

For example, imagine it’s 9am and today’s deadline is currently 10pm. I can change it to 9pm and the change will be immediate. If I try to change it to anything earlier than 9pm then then new deadline won’t take effect till the next day. And if it’s after 10am with a 10pm deadline then I can’t make any immediate changes to the deadline, including pushing it further out.

[3] Why 12 hours instead of 24? Because if you just passed the deadline and thought it sucked you should be able to change tomorrow’s deadline. But since you just passed today’s deadline, tomorrow’s deadline is within 24 hours. So we need a window and the simplest one is a full 12 hours.

[4] This was so WTF maybe we won’t even explain it, but this is a placeholder in case we decide to… But the important thing is that the long national nightmare is finally over!


 

Appendix: The Worse-Is-Better vs The-Right-Thing Debate

We had an interesting debate between the above worse-is-better choice where we minimize complications and special cases by just always treating the deadline as the day boundary, and The Right Thing which would be to respect the sanctity of the data and never mislead users about what happened when. For posterity, this was the tentivate alternative spec.

The Right Thing: QS Purity

We need to make a strong distinction between a datapoint on the 31st before the deadline and a datapoint on the 31st after the deadline. Users can’t be expected to understand that a datapoint that actually happened on one day is recorded as if it happened the following day just because it was after the deadline. The deadline is purely an aspect of the commitment device and can’t impact the QS aspect. Data happens when it happens and can’t be fudged.

Except night-owl deadlines. In that case it’s still the previous day until the deadline, even if that’s the next morning. Datapoints are recorded and plotted that way since that’s what users expect. Only for earlybird deadlines (before midnight) do we need a special case for plotting and recording the datapoint on the day it happened but marking it as having missed the deadline.

The distinction will be made visually on the graph and in the list of datapoints, like having a sad-face icon by each point that missed the deadline for that day.

Commentary: The unfortunate inconsistency here is how we have to treat the earlybird and night-owl cases differently. Night-owl goals have datapoints bucketed from today’s deadline to tomorrow’s deadline, while earlybird goals have datapoints bucketed from midnight to midnight. The special cases and implementation difficulties are in how we distinguish for the user between datapoints before and after the earlybird deadline on a given day.