-
Notifications
You must be signed in to change notification settings - Fork 8
/
README
343 lines (253 loc) · 14 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
A blessed fork of https://bitbucket.org/akoha/django-lean with a new maintainer.
Contact @jdunck on twitter if you need support, or the django-lean Google Group.
Existing, known issues, are here: https://bitbucket.org/akoha/django-lean/issues?status=new&status=open but please report new issues here on Github.
<<toc Home>>
== Welcome ==
{{{django-lean}}} aims to be a collection of tools for [[http://www.startuplessonslearned.com/search/label/lean%20startup|Lean Startups]] using the Django platform. Currently it provides a framework for implementing [[http://www.startuplessonslearned.com/search/label/split-test|split-test experiments]] in JavaScript, Python, or Django template code along with administrative views for analyzing the results of those experiments.
{{{django-lean}}} is one of the more mature AB testing systems for Django, but if you have contributions or other improvements, they're welcome.
== Discussion Group ==
For discussions related to the use or development of {{{django-lean}}} please use our [[http://groups.google.com/group/django-lean|Google group]].
== Features ==
{{{django-lean}}} allows you to perform split-test experiments on your users. In brief, this involves exposing 50% of your users to one implementation and 50% to another, then comparing the performance of these two groups with regards to certain metrics. Multi-variate testing (that is, a single experiment with more than 2 tested outcomes) are not yet supported, but we'd welcome changes to support that.
Often you can achive the goals of multivariate testing through concurrent experiments with the same control but different treatments, or through iteration of winner-as-control vs new candidate treatmemnt.
=== Experiments Types ===
{{{django-lean}}} supports two kinds of experiments currently:
* Anonymous Conversion experiments compare the achievement of goals you define (i.e. "register" or "add to cart") amongst two groups of anonymous users.
* Registered Engagement experiments compare a quantitative measure of engagement that you define (i.e., activity, revenue, time on site, ...) amongst two groups of registered users.
There's no real reason why one couldn't measure engagement of anonymous users or conversions of registered users (i.e. "basic to pro") but we didn't need this, so they're not implemented (again, patches welcome!).
=== Experiment Reports ===
{{{django-lean}}} provides daily reports of experiment results, including confidence levels.
* For conversion experiments, results and confidence are displayed per conversion goal type (and for 'any' goal). Confidence is calculated using the chi-square method.
* For engagement experiments, confidence is calculated using the Student's t-test method.
Experiment reports are prepared using the {{{update_experiment_reports}}} management command. It's advisable to execute this command from a nightly cron-job.
=== Bot Exclusion ===
{{{django-lean}}} attempts to exclude non-human visitors from experiment reports by only recording data for visitors who have JavaScript enabled.
=== Experiment Management ===
Experiments may be defined, enabled, disabled, or promoted via the {{{django-admin}}} interface. You may also define experiments in your source tree and have them automatically loaded into the database (see [[https://github.com/votizen/django-lean/blob/master/django_lean/experiments/loader.py|{{{experiments.loader.ExperimentLoader}}}]]).
Each experiment has a state, which affects whether visitors are enrolled in the experiment, and whether they see the control or test case of the experiment.
* {{{disabled}}}: No visitors are enrolled in the experiment. All visitors see the //control// case of the experiment, even if they were previously enrolled in the test group.
* {{{enabled}}}: All visitors who encounter the experiment are enrolled randomly in either the test or control group, and see the corresponding case.
* {{{promoted}}}: No visitors are enrolled in the experiment. All visitors see the //test// case of the experiment, even if they were previously enrolled in the control group.
New experiments start in the {{{disabled}}} state.
=== Experiment Implementation ===
{{{django-lean}}} makes it easy to implement experiments in Python, JavaScript, or Django templates. Here are some examples:
==== Python ====
{{{
#!python
from experiments.models import Experiment
from experiments.utils import WebUser
...
def my_view_func(request, *args, **kwargs):
if Experiment.test("my_experiment_name", WebUser(request)):
view = edit_profile_test
else:
view = edit_profile_control
return view(request, *args, **kwargs)
}}}
==== Django Templates ====
{{{
#!html+django
{% load experiments %}
...
<p>If you like what you see,
{% experiment change_buy_to_italics control %}
<b>buy now!</b>
{% endexperiment %}
{% experiment change_buy_to_italics test %}
<i>buy now!</i>
{% endexperiment %}
</p>
}}}
==== JavaScript ====
//(In your HTML template:)//
{{{
#!html+django
{% load experiments %}
<script>
<![CDATA[
{% include "experiments/include/experiments.js" %}
]]>
</script>
{% include "experiments/include/experiment_enrollment.html" %}
...
{% clientsideexperiment <experiment_name> %}
}}}
//(In your JavaScript:)//
{{{
#!javascript
if (experiment.test("<experiment_name>")) {
// test case
} else {
// control case
}
}}}
=== Conversion Tracking ===
Conversion experiments track the rate of conversion for their test and control groups. It is up to you to define and record the achievement of one or more project specific conversion goals.
==== Defining Conversion Goals ====
Conversion goals are defined by placing rows in the {{{experiments_goaltypes}}} table. This table is not currently exposed via {{{django-admin}}} but probably should be (patches welcome!). Alternative ways to populate it include manually via SQL, manually via the Django shell, via your {{{initial_data}}} fixture, or by defining a data migration in your database management tool (we use [[http://south.aeracode.org/|{{{django-south}}}]]).
Here is an example of defining a goal type using the Django shell:
{{{
#!pycon
erik-wrights-macbook-pro:akoha erikwright$ ./manage.py shell --plain
Python 2.6.2 (r262:71600, Jul 16 2009, 12:11:28)
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from experiments.models import GoalType
>>> GoalType.objects.create(name="signup")
<GoalType: <GoalType: signup>>
>>>
}}}
==== Recording Conversions ====
Once you have defined a conversion goal type, you must record it when it is achieved place. This may be done either programatically, or using a tracking pixel.
===== Recording Conversions Programatically =====
{{{
#!python
from experiments.models import GoalRecord
from experiments.utils import WebUser
...
def my_signup_view_func(request, *args, **kwargs):
# ... process the signup request
GoalRecord.record("signup", WebUser(request))
# ...
}}}
==== Recording Conversions Using a Tracking Pixel ====
Sometimes a conversion happens somewhere that you don't control (for example, an e-commerce platform). In this case, you can record the conversion by placing a transparent 1x1 pixel on the page that users see after the conversion occurs (i.e., the 'Thank You' page after purchase).
{{{
#!html
<img src="http://example.com/experiments/goal/purchase" height="1" width="1" />
}}}
=== Engagement Tracking ===
Engagement experiments track an arbitrary engagement value for each user in their test and control groups. It is up to you to define a function that calculates an appropriate engagement value for your users.
Here is an example engagement calculator:
{{{
#!python
class MyEngagementScoreCalculator(object):
def calculate_user_engagement_score(self, user, start_date, end_date):
"""
Define engagement as 'dollars spent per day'
"""
days_in_period = (end_date - start_date).days + 1
period_purchase_total = sum([p.subtotal for p in Purchase.objects.filter(
user=user, date__gte=start_date, date__lte=end_date)])
engagement_score = ((float)(period_purchase_total) /
days_in_period)
return engagement_score
}}}
Your engagement calculator must be registered in {{{settings.py}}} as follows:
{{{
#!python
...
LEAN_ENGAGEMENT_CALCULATOR = "mycompany.MyEngagementScoreCalculator"
...
}}}
== Dependencies ==
{{{django-lean}}} has a number of dependencies:
* [[http://code.google.com/p/pymox/|Mox]]
* [[http://www.crummy.com/software/BeautifulSoup/|Beautiful Soup]]
* [[http://jquery.com/|JQuery]]
Mox and Beautiful Soup are used exclusively by unit tests. JQuery is used only to execute a single, trivial AJAX request and could easily be removed from the dependency list if one were motivated (patch please!).
You may optionally use [[http://south.aeracode.org/|South]] in order to facilitate migrations of the {{{django-lean}}} database schema, but it is not required:
== Installation ==
{{{django-lean}}} has been developed with Django 1.0. Unit Tests run successfully with Django 1.1 but it has not been tried in production. If you successfully run it with another version, please update this documentation.
# Install {{{django-lean}}} using {{{easy_install}}}
# Add {{{experiments}}} to {{{INSTALLED_APPS}}} in {{{settings.py}}}
# Ensure that {{{django.core.context_processors.request}}} is in {{{TEMPLATE_CONTEXT_PROCESSORS}}} in {{{settings.py}}}
# Run {{{manage.py syncdb}}} to set up the {{{django-lean}}} tables.
# Run {{{manage.py test experiments}}} to see if everything is set up correctly.
# For every page that will contain an experiment (or in the response after a server-side experiment):
## Ensure that JQuery is included.
## Ensure that {{{experiments/include/experiments.js}}} is somehow included (perhaps copy it where your static files go, include it as part of your existing generated JS files, map it from {{{urls.py}}}, include it directly in a {{{<script/>}}} tag, etc.).
## Ensure that {{{experiments/include/experiment_enrollment.html}}} is rendered by your template.
# Install the admin and public url mappings in your site {{{urls.py}}}
# Register your engagement calculator in {{{settings.py}}}.
# Define one or more conversion goal types.
# Add conversion goal recording where appropriate.
# Define, implement, and enable an experiment
# Call {{{manage.py update_experiment_reports}}} nightly.
# Experiment, learn, repeat!
=== Installing URL Mappings ===
The following snippet added to {{urls.py}} should properly install the needed URL mappings (adjust to meet your needs):
{{{
#!python
...
urlpatterns += patterns('',
url(r'^admin/django-lean/', include('experiments.admin_urls')),
url(r'^django-lean/', include('experiments.urls')),
)
...
}}}
== Development ==
{{{
#!console
erik-wrights-macbook-pro:~ erikwright$ hg clone http://bitbucket.org/akoha/django-lean/
destination directory: django-lean
requesting all changes
adding changesets
adding manifests
adding file changes
added 46 changesets with 155 changes to 69 files
updating working directory
65 files updated, 0 files merged, 0 files removed, 0 files unresolved
erik-wrights-macbook-pro:~ erikwright$ cd django-lean
erik-wrights-macbook-pro:django-lean erikwright$ /usr/bin/python bootstrap.py
Creating directory '/Users/erikwright/django-lean/bin'.
Creating directory '/Users/erikwright/django-lean/parts'.
Creating directory '/Users/erikwright/django-lean/eggs'.
Creating directory '/Users/erikwright/django-lean/develop-eggs'.
Generated script '/Users/erikwright/django-lean/bin/buildout'.
erik-wrights-macbook-pro:django-lean erikwright$ ./bin/buildout
Develop: '/Users/erikwright/django-lean/.'
package init file 'experiments/tests/data/__init__.py' not found (or not a regular file)
Getting distribution for 'djangorecipe'.
Got djangorecipe 0.20.
Getting distribution for 'zc.recipe.egg'.
Got zc.recipe.egg 1.2.2.
Installing django.
django: Downloading Django from: http://www.djangoproject.com/download/1.1.1/tarball/
Getting distribution for 'mox'.
zip_safe flag not set; analyzing archive contents...
Got mox 0.5.0.
Getting distribution for 'BeautifulSoup'.
Got BeautifulSoup 3.1.0.1.
Generated script '/Users/erikwright/django-lean/bin/django'.
Generated script '/Users/erikwright/django-lean/bin/test'.
erik-wrights-macbook-pro:django-lean erikwright$ ./bin/test
Creating test database...
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table experiments_anonymousvisitor
Creating table experiments_goaltype
Creating table experiments_goalrecord
Creating table experiments_experiment
Creating table experiments_participant
Creating table experiments_dailyengagementreport
Creating table experiments_dailyconversionreport
Creating table experiments_dailyconversionreportgoaldata
Installing index for auth.Permission model
Installing index for auth.Message model
Installing index for experiments.AnonymousVisitor model
Installing index for experiments.GoalRecord model
Installing index for experiments.Experiment model
Installing index for experiments.Participant model
Installing index for experiments.DailyEngagementReport model
Installing index for experiments.DailyConversionReport model
Installing index for experiments.DailyConversionReportGoalData model
..............................
----------------------------------------------------------------------
Ran 30 tests in 24.305s
OK
Destroying test database...
erik-wrights-macbook-pro:django-lean erikwright$
}}}
== Other Resources ==
The following links might be of interest to those wanting to learn more about Lean Startup, split testing, and related concepts.
* http://elem.com/~btilly/effective-ab-testing/
* http://www.startuplessonslearned.com/search/label/lean%20startup
* http://www.startuplessonslearned.com/search/label/split-test
* http://www.slideshare.net/erikwright/djangolean-akohas-opensource-ab-experimentation-framework-montreal-python-9