Coding Rater

Total Resources Needed: 30 Expert Raters

Location: Remote (Dallas Texas)

Responsibilities:

• Model Quality Assessment: Evaluate the quality of AI model responses, including code, machine learning, and AI outputs. Identify errors, inefficiencies, and non-compliance with established standards.
• Code Annotation and Labeling: Accurately generate, annotate, and label code snippets, algorithms, and technical documentation according to project-specific guidelines.
• Review and Feedback: Provide detailed, constructive feedback on model and other outputs.
• Comparative Analysis: Compare multiple outputs and rank them based on criteria such as correctness, efficiency, readability, and adherence to programming best practices.
• Data Validation: Validate and correct datasets to ensure high-quality data for model training and evaluation.
• Collaboration: Work closely with data scientists and engineers to identify new annotation guidelines, resolve ambiguities, and contribute to the overall project strategy.

Qualifications:

• Strong background in software engineering/development, computer science, ML/AI, or related technical field, with a keen eye for detail and a passion for data accuracy.
• Programming Proficiency: Demonstrated expertise in:
  • Python (must-have)
  • At least one or more common programming languages such as: JavaScript, Rust, Node.js, Typescript, C, C++, Shell
  • (Bonus points) At least 1 or more less common programming languages such as: Rust, Shell, Go, Ruby, Swift, PHP, Kotlin
• Knowledge of Web Technologies & Frameworks:
  • Web Scraping, API integration
  • HTML/CSS/JavaScript
  • Web application development (e.g. Flask)
  • Frontend (e.g. React) and backend (e.g. Node.js) development
• Machine Learning & Artificial Intelligence:
  • Machine Learning (General concepts, model development, experimentation, training, evaluation)
  • Deep Learning (General, frameworks like TensorFlow, PyTorch, JAX, Keras, neural networks, CNNs, RNNs, transformer architecture, LSTM)
  • Natural Language Processing (NLP)
  • Reinforcement Learning (e.g., PPO, Q-learning, policy gradients, A2C, DQN, AlphaZero)
  • Computer Vision (e.g., image processing, analysis, instance segmentation, OCR, deepfake detection)
  • Game AI (Specific AI for intelligent opponents, understanding game states, actions, rewards)
• Data Science & Engineering:
  • Data Analysis & Manipulation (including Pandas, Matplotlib, Seaborn, NumPy, statistical analysis, general data processing, visualization libraries)
  • Database Management (SQL, NoSQL, SQLite, data storage)
• Algorithms & Mathematics:
  • Algorithms (General and specific like Monte Carlo Tree Search (MCTS), A* pathfinding, Sudoku solving, Collatz sequence, optimization, combinatorial problems)
• Software Engineering Practices & Tools:
  • Version Control (Git/GitHub)
  • Coding Best Practices: A solid understanding of clean code principles, software design patterns, and debugging techniques.
• Attention to Detail: Meticulous attention to detail and the ability to follow complex, multi-step instructions precisely.
• Problem-Solving: Strong analytical and problem-solving skills to evaluate and troubleshoot complex coding solutions.
• Communication: Excellent written communication skills to provide clear, concise, and actionable feedback.
• Proactiveness: Willingness to challenge the status quo to achieve the end goal.

Preferred Qualifications:

• Experience with AI/ML concepts, particularly with large language models (LLMs) and code generation.
• Familiarity with various programming paradigms (e.g., object-oriented, functional).
• Experience with code review in a professional or academic setting.
• Experience in data annotation or similar quality assurance roles.

Job Type: Contract

Pay: $15.00 - $20.00 per hour

Work Location: Remote