Official Tristan Language project compiler
Go to file
Tristan B. Velloza Kildaire ee537f2b25
Feature: Lexer improvements (#18)
* Created brsnch

* Created brsnch (removed placeholder)

* Lexer

- Added tab handling for the presence such as spaces would be in.
- Added unit tests for the new tab processing
- Resolved issues where whitepsace was allowed before and or after the
  '.' character
- Renamed isSpliter to isSplitter
- Some Code styling

* Check

- Added two new `SymbolType`s for comments
- `SINGLE_LINE_COMMENT` (for `//`) and `MULTI_LINE_COMMENT` (for `/*`)

* Parser

- Added a bogus `parseComment()` which returns nothing, prints out the comment, consumes the `Token` and returns
- `parseStatement()` now supports `parseComment()` whenever a single-line or multi-line comment is detected

* Parser

- Fixed token consumption code in `parseComment()`

* BasicLexer

- Fixed style mishaps

* ArrLexer

- Implemented dummy lexer

* Parser

- Added some comment related functions (for testing)
- Added `pushComment(Token)`, `hasCommentsOnStack() and `getCommentCount()`
- `parseComment()` now pushes the current comment-based `Token` onto the comment-stack
- Added a comment stack

Unit tests

- Added testing for various examples of comment-type `Token`s

* Lexer
- Replaced the characters with Enumerated type
- Working Comment lexing, single and multiline
- Working escape codes for strings
- Working Signage and Size Encoder indicators

- Removed floatMode in favour of float lexing function
- Added doComment for the comment lexing instead of comment mode
- Added doEscapeCode for escape codes in string

Testing
- Added unit tests for comments
- Added unit tests numerical encoders
- Added unit tests numerical encoders

TODO
- ADD unit tests for all valid escape sequences and some invalid

* Lexer
- Removed stringMode in favour of soString

TODO
- Decide on miltiline strings, currently not supported

* Parser

- Test comments which appear at a non-Module but rather statement lavel

* Parser

- Changed to using `BasicLexer` for comment tests now seeing as it is now implemented therein

* Basic

- Added `roll()` and `shourt()` to mark unittests

* Basic

- `shout()` now adds line number to print out

* Lexer rewrite
- flush
- underscores in numbers
- escape codes
- character escapes
- bug fixes

* Basic

- Fixed `shourt(int)`

* Basic

- Remved crashing (on purpose_ unittest

* Resolved bug where isSplitter evaluated to true every time

* Basic

- Removed `goggaWithLineInfo(...)`

* Basic

- Updated `shout()` to remove rolling
- Removed `roll()`
- Added function and module name as well

* Basic

- Documented `shout()`

* Lexer Done and 100% coverage

* LexerSymbols

- Documented
- Formatted

* Lexer (module)

- Added `LS` alias
- Added `isOperator(char c)`, `isSplitter(char c)`, `isNumericalEncoder_Size(char character)`, `isNumericalEncoder_Signage(char character)` and `isValidEscape_String(char character)`

* BasicLexer

- Documented constructor `hasToken()`, `performLex()`, `doIdentOrPath()`, `doChar()`, `doString()`, `doComment()`, `doEscapeCode()`, `doNumber()`, `doEncoder()`, `doFloat()`, `flush()`, `buildAdvance()`, `improvedAdvance()`, `advanceLine()`, `isOperator(char)`, `isSplitter(char)`, `isValidDotPrecede(char character)`, `isNumericalEncoder(char character)`, `isNumericalEncoder_Size(char character)`, `isNumericalEncoder_Signage(char character)` and `isValidEscape_String(char character)`
- Tried reformatting some of `doChar()`, `doString()`, `flush()`, `buildAdvance()`, `improvedAdvance()`, `advanceLine()`, `isOperator(char)`, `isSplitter(char)`

* Basic

- Removed `LS` alias

Lexer

- Made `LS` alias public

* BasicLexer

- Removed methods `isValidEscape_String(char character)`, `isNumericalEncoder_Signage(char character)`, `isNumericalEncoder_Size(char character)`, `isNumericalEncoder(char character)`, `isSplitter(char c)` and ` isOperator(char c)`

Lexer

- Added method `isNumericalEncoder(char character)`

* BasicLexer

- Documented `isValidDotPrecede(char character)`

* Lexer

- Added method `isValidDotPrecede(char character)`

* BasicLexer

- Removed method `isValidDotPrecede(char character)`

* BasicLexer (unittests)

- Documented the unittests
- Fixed formatting

* BasicLexer

- Typo fixes

* BasicLexer (unittests)

- Only compile-in `shourt(...)` when in unittest build mode

* BasicLexer

- Documented `isForward()` and `isBackward()`
- Made `isBackward()` private

---------

Co-authored-by: GMeyer <21568499@sun.ac.za>
Co-authored-by: GMeyer <gustav.meyer1999@gmail.com>
2023-12-27 08:18:17 +02:00
.github/workflows Pipelines 2023-12-08 18:27:42 +02:00
branding Array support (#1) 2023-04-20 11:21:50 +02:00
rando Fuck this is complicated (2/2) 2021-06-06 22:46:06 +02:00
source/tlang Feature: Lexer improvements (#18) 2023-12-27 08:18:17 +02:00
.gitignore 🐞️ Functions: Expressionless return and enforcing requirement (#7) 2023-07-11 21:43:21 +02:00
README.md Update README.md 2023-08-21 14:41:22 +02:00
dub.json Dub 2023-08-14 22:12:28 +02:00
extern_test.sh Check 2023-01-28 18:12:49 +02:00
macro_ast_replacemenet.png 🧠 Feature: Meta-programming engine (#10) 2023-05-29 17:02:28 +02:00
macro_ast_replacemenet.xcf 🧠 Feature: Meta-programming engine (#10) 2023-05-29 17:02:28 +02:00
macro_ast_replacement_sizeof_example.png 🧠 Feature: Meta-programming engine (#10) 2023-05-29 17:02:28 +02:00
macro_ast_replacement_sizeof_example.xcf 🧠 Feature: Meta-programming engine (#10) 2023-05-29 17:02:28 +02:00
malloc_test.sh Pointer support (#2) 2023-04-17 16:50:11 +02:00
tets.d Added testing files 2021-11-02 15:14:03 +02:00
texterned.c - Added `texterned.c` for later extern testing 2023-01-19 17:52:46 +02:00
tlang 🧠️ Feature: Universal coercion and type enforcer (#9) 2023-08-10 19:42:11 +02:00
todo1.md I wonder if that worked 2021-06-07 14:27:36 +02:00
wip.txt Array support (#1) 2023-04-20 11:21:50 +02:00

README.md

tlang

D Coverage Status

Official Tristan Language project compiler

Documentation

Docs are available here.

Building

To build you will need dmd and dub installed. You can then run the following:

dub test --coverage
dub build

The first command is optional but you want to make sure the compiler is working so you may as well run all the unit tests - it won't take too long.

Usage

A list of all commands can be found by typing in the following:

./tlang