Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[STUD-406] Fix/cdodge/unescape quote strings #387

Merged
merged 6 commits into from
Jul 12, 2013
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions common/lib/xmodule/xmodule/tests/test_xml_module.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from xmodule.x_module import XModuleFields
from xblock.core import Scope, String, Dict, Boolean, Integer, Float, Any, List
from xmodule.fields import Date, Timedelta
from xmodule.xml_module import XmlDescriptor, serialize_field, deserialize_field
from xmodule.xml_module import XmlDescriptor, serialize_field, deserialize_field, serialize_string_literal
import unittest
from .import get_test_system
from nose.tools import assert_equals
Expand Down Expand Up @@ -186,12 +186,27 @@ def test_serialize(self):
assert_equals('"false"', serialize_field('false'))
assert_equals('"fAlse"', serialize_field('fAlse'))
assert_equals('"hat box"', serialize_field('hat box'))
assert_equals('{"bar": "hat", "frog": "green"}', serialize_field({'bar': 'hat', 'frog' : 'green'}))
assert_equals('{"bar": "hat", "frog": "green"}', serialize_field({'bar': 'hat', 'frog': 'green'}))
assert_equals('[3.5, 5.6]', serialize_field([3.5, 5.6]))
assert_equals('["foo", "bar"]', serialize_field(['foo', 'bar']))
assert_equals('"2012-12-31T23:59:59Z"', serialize_field("2012-12-31T23:59:59Z"))
assert_equals('"1 day 12 hours 59 minutes 59 seconds"',
serialize_field("1 day 12 hours 59 minutes 59 seconds"))
serialize_field("1 day 12 hours 59 minutes 59 seconds"))

def test_serialize_string_literal(self):
assert_equals('2', serialize_string_literal('2'))
assert_equals('2.589', serialize_string_literal('2.589'))
assert_equals('false', serialize_string_literal('false'))
assert_equals('fAlse', serialize_string_literal('fAlse'))
assert_equals('hat box', serialize_string_literal('hat box'))
assert_equals('2012-12-31T23:59:59Z', serialize_string_literal("2012-12-31T23:59:59Z"))
assert_equals('1 day 12 hours 59 minutes 59 seconds',
serialize_string_literal("1 day 12 hours 59 minutes 59 seconds"))

try:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need this try around the self.assertRaises.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The correct syntax for assertRaises is:

with self.assertRaises(Exception):
    serialize_string_literal(2.31)

or, less preferred: self.assertRaises(Exception, serialize_string_literal, 2.31)

self.assertRaises(serialize_string_literal(2.31))
except Exception:
pass


class TestDeserialize(unittest.TestCase):
Expand Down
54 changes: 31 additions & 23 deletions common/lib/xmodule/xmodule/xml_module.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,13 +81,23 @@ def __new__(_cls, from_xml=lambda x: x,

def serialize_field(value):
"""
Return a string version of the value (where value is the JSON-formatted, internally stored value).
Return a string version of the value (where value is the JSON-formatted, internally stored value).

By default, this is the result of calling json.dumps on the input value.
"""
By default, this is the result of calling json.dumps on the input value.
"""
return json.dumps(value, cls=EdxJSONEncoder)


def serialize_string_literal(value):
"""
Assert that the value is a base string and - if it is - simply return it
"""
if not isinstance(value, basestring):
raise Exception('Value {0} is not of type basestring!'.format(value))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a TypeError, not a base Exception


return value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd rather have this logic just live in serialize_field, rather than having a lambda containing an if clause in the AttrMap

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe @cahrens can comment on this but I wasn't sure if we want this behavior globally or just on the 'export' feature. That's why I separated the helper functions.

If serialize_field is current - and remain so in the future - only called on export use cases, then I'm fine with moving the logic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serialize_field should really only be for serializing fields to xml, which is exactly when we want this behavior.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Christina, can you confirm this? If so, then I can refactor this - as well
as updating the existing test cases.

On Fri, Jul 12, 2013 at 12:22 PM, Calen Pennington <notifications@github.com

wrote:

In common/lib/xmodule/xmodule/xml_module.py:

 return json.dumps(value, cls=EdxJSONEncoder)

+def serialize_string_literal(
value):

  • """
  • Assert that the value is a base string and - if it is - simply return it
  • """
  • if not isinstance(value, basestring):
  •    raise Exception('Value {0} is not of type basestring!'.format(value))
    
  • return value

serialize_field should really only be for serializing fields to xml,
which is exactly when we want this behavior.


Reply to this email directly or view it on GitHubhttps://github.com/edx/edx-platform/pull/387/files#r5168720
.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree this should be within serialize_field. It will also make the unit tests represent how we actually serialize strings.



def deserialize_field(field, value):
"""
Deserialize the string version to the value stored internally.
Expand Down Expand Up @@ -126,7 +136,7 @@ class XmlDescriptor(XModuleDescriptor):
"""

xml_attributes = Dict(help="Map of unhandled xml attributes, used only for storage between import and export",
default={}, scope=Scope.settings)
default={}, scope=Scope.settings)

# Extension to append to filename paths
filename_extension = 'xml'
Expand All @@ -141,23 +151,23 @@ class XmlDescriptor(XModuleDescriptor):
# understand? And if we do, is this the place?
# Related: What's the right behavior for clean_metadata?
metadata_attributes = ('format', 'graceperiod', 'showanswer', 'rerandomize',
'start', 'due', 'graded', 'display_name', 'url_name', 'hide_from_toc',
'ispublic', # if True, then course is listed for all users; see
'xqa_key', # for xqaa server access
'giturl', # url of git server for origin of file
# information about testcenter exams is a dict (of dicts), not a string,
# so it cannot be easily exportable as a course element's attribute.
'testcenter_info',
# VS[compat] Remove once unused.
'name', 'slug')
'start', 'due', 'graded', 'display_name', 'url_name', 'hide_from_toc',
'ispublic', # if True, then course is listed for all users; see
'xqa_key', # for xqaa server access
'giturl', # url of git server for origin of file
# information about testcenter exams is a dict (of dicts), not a string,
# so it cannot be easily exportable as a course element's attribute.
'testcenter_info',
# VS[compat] Remove once unused.
'name', 'slug')

metadata_to_strip = ('data_dir',
'tabs', 'grading_policy', 'published_by', 'published_date',
'discussion_blackouts', 'testcenter_info',
# VS[compat] -- remove the below attrs once everything is in the CMS
'course', 'org', 'url_name', 'filename',
# Used for storing xml attributes between import and export, for roundtrips
'xml_attributes')
'tabs', 'grading_policy', 'published_by', 'published_date',
'discussion_blackouts', 'testcenter_info',
# VS[compat] -- remove the below attrs once everything is in the CMS
'course', 'org', 'url_name', 'filename',
# Used for storing xml attributes between import and export, for roundtrips
'xml_attributes')

metadata_to_export_to_policy = ('discussion_topics')

Expand All @@ -166,7 +176,7 @@ def get_map_for_field(cls, attr):
for field in set(cls.fields + cls.lms.fields):
if field.name == attr:
from_xml = lambda val: deserialize_field(field, val)
to_xml = lambda val : serialize_field(val)
to_xml = lambda val: serialize_string_literal(val) if isinstance(val, basestring) else serialize_field(val)
return AttrMap(from_xml, to_xml)

return AttrMap()
Expand Down Expand Up @@ -254,7 +264,7 @@ def load_definition(cls, xml_object, system, location):
definition, children = cls.definition_from_xml(definition_xml, system)
if definition_metadata:
definition['definition_metadata'] = definition_metadata
definition['filename'] = [ filepath, filename ]
definition['filename'] = [filepath, filename]

return definition, children

Expand All @@ -280,7 +290,6 @@ def load_metadata(cls, xml_object):
metadata[attr] = attr_map.from_xml(val)
return metadata


@classmethod
def apply_policy(cls, metadata, policy):
"""
Expand Down Expand Up @@ -374,7 +383,6 @@ def export_to_file(self):
"""
return True


def export_to_xml(self, resource_fs):
"""
Returns an xml string representing this module, and all modules
Expand Down
8 changes: 4 additions & 4 deletions common/test/data/simple/course.xml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<course name="A Simple Course" org="edX" course="simple" graceperiod="1 day 5 hours 59 minutes 59 seconds" slug="2012_Fall">
<chapter name="Overview">
<video name="Welcome" youtube_id_0_75="&quot;izygArpw-Qo&quot;" youtube_id_1_0="&quot;p2Q6BrNhdh8&quot;" youtube_id_1_25="&quot;1EeWXzPdhSA&quot;" youtube_id_1_5="&quot;rABDYkeK0x8&quot;"/>
<video name="Welcome" youtube_id_0_75="izygArpw-Qo" youtube_id_1_0="p2Q6BrNhdh8" youtube_id_1_25="1EeWXzPdhSA" youtube_id_1_5="rABDYkeK0x8"/>
<videosequence format="Lecture Sequence" name="A simple sequence">
<html name="toylab" filename="toylab"/>
<video name="S0V1: Video Resources" youtube_id_0_75="&quot;EuzkdzfR0i8&quot;" youtube_id_1_0="&quot;1bK-WdDi6Qw&quot;" youtube_id_1_25="&quot;0v1VzoDVUTM&quot;" youtube_id_1_5="&quot;Bxk_-ZJb240&quot;"/>
<video name="S0V1: Video Resources" youtube_id_0_75="EuzkdzfR0i8" youtube_id_1_0="1bK-WdDi6Qw" youtube_id_1_25="0v1VzoDVUTM" youtube_id_1_5="Bxk_-ZJb240"/>
</videosequence>
<section name="Lecture 2">
<sequential>
<video youtube_id_1_0="&quot;TBvX7HzxexQ&quot;"/>
<video youtube_id_1_0="TBvX7HzxexQ"/>
<problem name="L1 Problem 1" points="1" type="lecture" showanswer="attempted" filename="L1_Problem_1" rerandomize="never"/>
</sequential>
</section>
Expand All @@ -18,7 +18,7 @@
<problem type="lecture" showanswer="attempted" rerandomize="true" display_name="A simple coding problem" name="Simple coding problem" filename="ps01-simple" url_name="ps01-simple"/>
</sequential>
</section>
<video name="Lost Video" youtube_id_1_0="&quot;TBvX7HzxexQ&quot;"/>
<video name="Lost Video" youtube_id_1_0="TBvX7HzxexQ"/>
<sequential format="Lecture Sequence" url_name='test_sequence'>
<vertical url_name='test_vertical'>
<html url_name='test_html'>
Expand Down
4 changes: 2 additions & 2 deletions common/test/data/test_exam_registration/course/2012_Fall.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
<chapter url_name="Overview">
<videosequence url_name="Toy_Videos">
<html url_name="toylab"/>
<video url_name="Video_Resources" youtube_id_1_0="&quot;1bK-WdDi6Qw&quot;"/>
<video url_name="Video_Resources" youtube_id_1_0="1bK-WdDi6Qw"/>
</videosequence>
<video url_name="Welcome" youtube_id_1_0="&quot;p2Q6BrNhdh8&quot;"/>
<video url_name="Welcome" youtube_id_1_0="p2Q6BrNhdh8"/>
</chapter>
<chapter url_name="Ch2">
<html url_name="test_html">
Expand Down
4 changes: 2 additions & 2 deletions common/test/data/test_start_date/course/2012_Fall.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
<chapter url_name="Overview">
<videosequence url_name="Toy_Videos">
<html url_name="toylab"/>
<video url_name="Video_Resources" youtube_id_1_0="&quot;1bK-WdDi6Qw&quot;"/>
<video url_name="Video_Resources" youtube_id_1_0="1bK-WdDi6Qw"/>
</videosequence>
<video url_name="Welcome" youtube_id_1_0="&quot;p2Q6BrNhdh8&quot;"/>
<video url_name="Welcome" youtube_id_1_0="p2Q6BrNhdh8"/>
</chapter>
<chapter url_name="Ch2">
<html url_name="test_html">
Expand Down
2 changes: 1 addition & 1 deletion common/test/data/toy/chapter/secret/magic.xml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
<chapter>
<video url_name="toyvideo" youtube_id_1_0="&quot;OEoXaMPEzfM&quot;"/>
<video url_name="toyvideo" youtube_id_1_0="OEoXaMPEzfM"/>
</chapter>
8 changes: 4 additions & 4 deletions common/test/data/toy/course/2012_Fall.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
<chapter url_name="Overview">
<videosequence url_name="Toy_Videos">
<html url_name="secret:toylab"/>
<video url_name="Video_Resources" youtube_id_1_0="&quot;1bK-WdDi6Qw&quot;"/>
<video url_name="Video_Resources" youtube_id_1_0="1bK-WdDi6Qw"/>
</videosequence>
<video url_name="Welcome" youtube_id_1_0="&quot;p2Q6BrNhdh8&quot;"/>
<video url_name="video_123456789012" youtube_id_1_0="&quot;p2Q6BrNhdh8&quot;"/>
<video url_name="video_4f66f493ac8f" youtube_id_1_0="&quot;p2Q6BrNhdh8&quot;"/>
<video url_name="Welcome" youtube_id_1_0="p2Q6BrNhdh8"/>
<video url_name="video_123456789012" youtube_id_1_0="p2Q6BrNhdh8"/>
<video url_name="video_4f66f493ac8f" youtube_id_1_0="p2Q6BrNhdh8"/>
</chapter>
<chapter url_name="secret:magic"/>
<chapter url_name="poll_test"/>
Expand Down
2 changes: 1 addition & 1 deletion common/test/data/two_toys/video/Video_Resources.xml
Original file line number Diff line number Diff line change
@@ -1 +1 @@
<video youtube_id_1_0="&quot;1bK-WdDi6Qw&quot;" display_name="Video Resources"/>
<video youtube_id_1_0="1bK-WdDi6Qw" display_name="Video Resources"/>