Rich text has been disabled in custom editor scripts by default. This has been done as there were some issues with custom editors showing scripts of users and making script tags to conflict with rich text tags. To enable rich text in editor scripts, you would have to first create a new GUIStyle:
GUIStyle style = new GUIStyle ();
//Enable rich text:
style.richText = true;
//And apply it with GUILayout.BeginArea() or use it for each GUI element:
GUILayout.Label("<size=30>Some <color=yellow>RICH</color> text</size>",style);